Systems and Methods for Enabling Low-Vision Users to Interact with a Touch-Sensitive Secondary Display

ABSTRACT

Disclosed herein are systems and methods that enable low-vision users to interact with touch-sensitive secondary displays. An example method includes, while operating a touch-sensitive secondary display in an accessibility mode: displaying, on the primary display, a first user interface for an application, and displaying, on the touch-sensitive secondary display, a second user interface that includes: (i) application-specific affordances, and (ii) a system-level affordance, where each application-specific affordance and the system-level affordance are displayed with a first display size. The method includes detecting an input at the application-specific affordance. In response to detecting the input, and while the input remains in contact: continuing to display the first user interface for the application; and displaying, on the primary display, a zoomed-in representation of the at least one application-specific affordance, where the zoomed-in representation of the application-specific affordance is displayed with a second display size that is larger than the first display size.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/791,251, filed Oct. 23, 2017, which claims priority to U.S.Provisional Application Ser. No. 62/412,752, filed Oct. 25, 2016, whichare herein incorporated by reference in their entireties.

TECHNICAL FIELD

The disclosed embodiments relate to touch-sensitive secondary displaydevices and, more specifically, to improved techniques for enablinglow-vision users to interact with touch-sensitive secondary displays.

BACKGROUND

Occlusion problems often prevent many users from appreciating and usingfeatures available through touch-sensitive input devices that are alsoused to display affordances. For example, users of touch-sensitivesecondary displays that may be located above a physical keyboard may notbe able to view certain affordances because their fingers are occludingor covering up the affordances while they are displayed at a secondarydisplay. Moreover, the affordances displayed in such secondary displaysare often small. These problems are particularly acute for low-visionusers, who may have difficulties seeing certain affordances that aredisplayed at a secondary display, and these difficulties are worsenedand amplified by the aforementioned occlusion problems.

SUMMARY

The embodiments described herein address the above shortcomings byproviding devices and methods that seamlessly offer zoomed-inrepresentations of affordances at a primary display, in response to userinteractions at a touch-sensitive secondary display device, therebyenabling low-vision users to interact with touch-sensitive secondarydisplays (i.e., these users are now able to view a zoomed-inrepresentation at the primary display and then determine appropriateinputs to provide at the secondary display). Such devices and methodsalso reduce the amount of mode switching (e.g., moving one's handsbetween keyboard and mouse, and also moving one's eyes from keyboard todisplay) required of a user and thereby reduce the number of inputsrequired to activate a desired function (e.g., number of inputs requiredto select menu options is reduced, as explained in more detail below).Such devices and methods also make more information available on alimited screen (e.g., a touch-sensitive secondary display is used toprovide more information to a user and this information is efficientlypresented using limited screen space). Such devices and methods alsoprovide improved man-machine interfaces, e.g., by providing emphasizingeffects to make information more discernable on a touch-sensitivesecondary display, by providing sustained interactions so thatsuccessive inputs from a user directed to either a touch-sensitivesecondary display or a primary display cause the device to provideoutputs which are then used to facilitate further inputs from the user(e.g., affordances are displayed at the touch-sensitive secondarydisplay that allow users to quickly preview how information will berendered on a primary display, by providing inputs at thetouch-sensitive secondary display, as discussed below), and by requiringfewer interactions from users to achieve desired results. In someinstances, the touch-sensitive secondary display is also referred toherein as a dynamic function row (and vice versa). For these reasons andthose discussed below, the devices and methods described herein reducepower usage and improve battery life of electronic devices.

In accordance with some embodiments, a method is performed at acomputing system with one or more processors, a first housing thatincludes a primary display, memory, and a second housing (that isdistinct from the first housing) at least partially containing atouch-sensitive secondary display that is distinct from the primarydisplay (as discussed below, the second housing and the touch-sensitivesecondary display may be components of any device that includes asmaller display than that of the primary display, e.g., thetouch-sensitive secondary display is part of a wearable computingdevice, such as a watch, or the touch-sensitive secondary display islocated above a physical keyboard in the second housing). The methodincludes: displaying, on the primary display, a first user interface foran application. The method also includes: displaying, on thetouch-sensitive secondary display, a second user interface that includesa plurality of application-specific affordances that control functionsavailable within the application, where each of the plurality ofapplication-specific affordances is displayed with a first display size.The method additionally includes: detecting, via the touch-sensitivesecondary display, an input that contacts at least oneapplication-specific affordance of the plurality of application-specificaffordances. In response to detecting the input and while the inputremains in contact with the touch-sensitive secondary display, themethod includes: (i) continuing to display, on the primary display, thefirst user interface for the application and, (ii) displaying, on theprimary display, a zoomed-in representation of the at least oneapplication-specific affordance, where the zoomed-in representation ofthe at least one application-specific affordance is displayed with asecond display size that is larger than the first display size.

In some instances, users of computing systems (in particular, low-visionusers) may be unable to accurately view icons or affordances that aredisplayed with a small display size (such as those shown on a smartwatch). Populating a touch-sensitive secondary display withapplication-specific affordances and then displaying a zoomed-inrepresentation of one of those affordances at a larger, primary displayin response to a single input provides these users with clear visualfeedback indicating which affordance they may be selecting. Providingthis improved visual feedback to the user enhances operability of thedevice and makes the human-machine interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with affordances displayed on the secondarydisplay). Additionally, allowing these users to accurately viewaffordances displayed on a small screen enables a sustained interactionwith the touch-sensitive secondary display that would not otherwise bepossible due to frequent mistakes (e.g., incorrect selections) and theadditional time and effort needed to correct those mistakes.

In accordance with some embodiments, a method is performed at acomputing system with one or more processors, a first housing thatincludes a primary display, memory, and a second housing (that isdistinct from the first housing) at least partially containing atouch-sensitive secondary display that is distinct from the primarydisplay (as discussed below, the second housing and the touch-sensitivesecondary display may be components of any device that includes asmaller display than that of the primary display, e.g., thetouch-sensitive secondary display is part of a wearable computingdevice, such as a watch, or the touch-sensitive secondary display islocated above a physical keyboard in the second housing). The methodincludes: operating the touch-sensitive secondary display in anaccessibility mode. While operating the touch-sensitive secondarydisplay in the accessibility mode, the method includes: displaying, onthe primary display, a first user interface for an application;displaying, on the touch-sensitive secondary display, a second userinterface that includes: (i) a plurality of application-specificaffordances that control functions available within the application, and(ii) at least one system-level affordance that controls a system-levelfunction, wherein each of the plurality of application-specificaffordances and the at least one system-level affordance are displayedwith a first display size; detecting, via the touch-sensitive secondarydisplay, an input that contacts at least one application-specificaffordance of the plurality of application-specific affordances; and inresponse to the detecting the input and while the input remains incontact with the touch-sensitive secondary display: (A) continuing todisplay, on the primary display, the first user interface for theapplication and (B) displaying, on the primary display, a zoomed-inrepresentation of the at least one application-specific affordance,wherein the zoomed-in representation of the at least oneapplication-specific affordance is displayed with a second display sizethat is larger than the first display size.

In some instances, low-vision users of computing systems rely onmemorized key locations on a keyboard so that they are able toaccurately provide inputs to a computing system. For computing systemsthat include a touch-sensitive secondary display with often-changingaffordances (e.g., affordances that change to provide functions that areuseful to a user based on what they are doing within a particularapplication), these users are not able to rely solely on memorization toprovide accurate inputs. Displaying a zoomed-in representation of atleast one affordance of the application-specific affordance improvesoperability of the computing system, because low-vision users are ableto interact with controls available at the touch-sensitive secondarydisplay that may be too small (or may be occluded from view because auser's finger is covering up the displayed controls) for the low-visionusers to view accurately. In this way, low-vision users are able to takeadvantage of an improved man-machine interface by, e.g., havingsustained interactions with a touch-sensitive secondary display (insteadof having to constantly correct erroneous inputs).

In accordance with some embodiments, a method is performed at acomputing system with one or more processors, a first housing thatincludes a primary display, memory, and a second housing (that isdistinct from the first housing) at least partially containing atouch-sensitive secondary display that is distinct from the primarydisplay (as discussed below, the second housing and the touch-sensitivesecondary display may be components of any device that includes asmaller display than that of the primary display, e.g., thetouch-sensitive secondary display is part of a wearable computingdevice, such as a watch, or the touch-sensitive secondary display islocated above a physical keyboard in the second housing). The methodincludes: displaying, on the primary display, a first user interface foran application. The method also includes: displaying, on thetouch-sensitive secondary display, a second user interface thatincludes: (i) a plurality of application-specific affordances thatcontrol functions available within the application and (ii) at least onesystem-level affordance that controls a system-level function. Themethod additionally includes: detecting, via the touch-sensitivesecondary display, a first input over a first application-specificaffordance of the plurality of application-specific affordances. Whilethe first input remains in contact with the first application-specificaffordance, the method includes: (i) detecting, via the touch-sensitivesecondary display, a second input that is not over the firstapplication-specific affordance and (ii) in response to detecting thesecond input, activating the first application-specific affordance.

Allowing activation of an affordance that is in contact with an input inresponse to a tap gesture that is not over the affordance (a “split-tapgesture”) enhances operability of the device and makes the human-machineinterface more efficient (e.g., by allowing users to place a firstfinger over a desired affordance and then use a different finger toperform a selection of that desired affordance, thereby ensuring thatonly the desired affordance is activated and helping to minimizeerroneous selections/activations). Additionally, allowing users to movetheir first finger freely around the touch-sensitive secondary display(without selecting affordances) allows users to maintain a sustainedinteraction with the touch-sensitive secondary display (by exploringwhich affordances are displayed at the touch-sensitive secondarydisplay), that would not otherwise be possible due to frequent mistakes(e.g., incorrect or accidental selections of affordances) and theadditional time and effort needed to correct those mistakes.

In accordance with some embodiments, a computing system includes a firsthousing with a primary display unit configured to display userinterfaces, a second housing (that is distinct from the first housing)at least partially containing a touch-sensitive secondary display thatis distinct from the primary display and that is configured to receiveuser inputs and to display user interfaces, and a processing unit thatis in communication with the primary display unit and thetouch-sensitive secondary display unit. The processing unit isconfigured to: display, on the primary display, a first user interfacefor an application; display, on the touch-sensitive secondary display, asecond user interface that includes a plurality of application-specificaffordances that control functions available within the application, andeach of the plurality of application-specific affordances is displayedwith a first display size; detect, via the touch-sensitive secondarydisplay, an input that contacts at least one application-specificaffordance of the plurality of application-specific affordances; and inresponse to detecting the input and while the input remains in contactwith the touch-sensitive secondary display: (i) continue to display, onthe primary display, the first user interface for the application and(ii) display, on the primary display, a zoomed-in representation of theat least one application-specific affordance, wherein the zoomed-inrepresentation of the at least one application-specific affordance isdisplayed with a second display size that is larger than the firstdisplay size.

In accordance with some embodiments, a computing system includes a firsthousing with a primary display unit configured to display userinterfaces, a second housing (that is distinct from the first housing)at least partially containing a touch-sensitive secondary display thatis distinct from the primary display and that is configured to receiveuser inputs and to display user interfaces, and a processing unit thatis in communication with the primary display unit and thetouch-sensitive secondary display unit. The processing unit isconfigured to: operate the touch-sensitive secondary display in anaccessibility mode; while operating the touch-sensitive secondarydisplay in the accessibility mode: display, on the primary display, afirst user interface for an application; display, on the touch-sensitivesecondary display, a second user interface that includes: (i) aplurality of application-specific affordances that control functionsavailable within the application and (ii) at least one system-levelaffordance that controls a system-level function, wherein each of theplurality of application-specific affordances and the at least onesystem-level affordance are displayed with a first display size; detect,via the touch-sensitive secondary display, an input that contacts atleast one application-specific affordance of the plurality ofapplication-specific affordances; and in response to the detecting theinput and while the input remains in contact with the touch-sensitivesecondary display: (A) continue to display, on the primary display, thefirst user interface for the application and (B) display, on the primarydisplay, a zoomed-in representation of the at least oneapplication-specific affordance, wherein the zoomed-in representation ofthe at least one application-specific affordance is displayed with asecond display size that is larger than the first display size.

In accordance with some embodiments, a computing system includes a firsthousing with a primary display unit configured to display userinterfaces, a second housing (that is distinct from the first housing)at least partially containing a touch-sensitive secondary display thatis distinct from the primary display and that is configured to receiveuser inputs and to display user interfaces, and a processing unit thatis in communication with the primary display unit and thetouch-sensitive secondary display unit. The processing unit isconfigured to: display, on the primary display, a first user interfacefor an application; display, on the touch-sensitive secondary display, asecond user interface that includes: (i) a plurality ofapplication-specific affordances that control functions available withinthe application and (ii) at least one system-level affordance thatcontrols a system-level function; detect, via the touch-sensitivesecondary display, a first input over a first application-specificaffordance of the plurality of application-specific affordances; andwhile the first input remains in contact with the firstapplication-specific affordance: (i) detect, via the touch-sensitivesecondary display, a second input that is not over the firstapplication-specific affordance and (ii) in response to detecting thesecond input, activate the first application-specific affordance.

In accordance with some embodiments, a computing system includes one ormore processors, a first housing with a primary display, a secondhousing at least partially containing a touch-sensitive secondarydisplay that is distinct from the primary display and optionallycontaining one or more sensors to detect intensity of contacts with thetouch-sensitive secondary surface, and memory storing one or moreprograms, the one or more programs configured for execution by the oneor more processors and the one or more programs include instructions forperforming or causing performance of the operations of any of themethods described herein. In accordance with some embodiments, acomputer-readable storage medium has stored therein instructions that,when executed by the computing system, cause the computing system toperform or cause performance of the operations of any of the methodsdescribed herein. In accordance with some embodiments, a graphical userinterface on the primary display of the computing system is provided,and the graphical user interface includes one or more of the elementsdisplayed in any of the methods described herein, which are updated inresponse to inputs, as described in any of the methods described herein.In accordance with some embodiments, the computing system includes meansfor performing or causing performance of the operations of any of themethods described herein. In accordance with some embodiments, aninformation processing apparatus, for use in the computing system,includes means for performing or causing performance of the operationsof any of the methods described herein.

Thus, computing systems that include both primary and touch-sensitivesecondary displays are provided with faster, more efficient andusable/user-friendly methods and interfaces for enabling low-visionusers to interact with affordances displayed at touch-sensitivesecondary displays, thereby improving operability of the computingsystem by, e.g., allowing users to have sustained interactions with thetouch-sensitive secondary display.

BRIEF DESCRIPTION OF DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1A is an illustrative diagram of a portable computing system (e.g.,a laptop computer), in accordance with some embodiments.

FIG. 1B is an illustrative diagram of a body portion of the portablecomputing system in FIG. 1A, in accordance with some embodiments.

FIG. 2A is an illustrative diagram of a first implementation of adesktop computing system, in accordance with some embodiments.

FIG. 2B is an illustrative diagram of a second implementation of adesktop computing system, in accordance with some embodiments.

FIG. 2C is an illustrative diagram of a third implementation of adesktop computing system, in accordance with some embodiments.

FIG. 2D is an illustrative diagram of a fourth implementation of adesktop computing system, in accordance with some embodiments.

FIG. 3A is a block diagram of an electronic device, in accordance withsome embodiments.

FIG. 3B is a block diagram of components for event handling of FIG. 3A,in accordance with some embodiments.

FIGS. 3C-3E illustrate examples of dynamic intensity thresholds inaccordance with some embodiments.

FIG. 4 is a block diagram of a peripheral electronic device, inaccordance with some embodiments.

FIGS. 5A-5N are schematics of primary and secondary displays used toillustrate example user interfaces for enabling low-vision users tointeract with touch-sensitive secondary displays, in accordance withsome embodiments.

FIGS. 6A-6C show a flowchart of a method of enabling low-vision users tointeract with touch-sensitive secondary displays, in accordance withsome embodiments.

FIGS. 7A-7C show a flowchart of a method of enabling low-vision users tointeract with touch-sensitive secondary displays, in accordance withsome embodiments.

FIGS. 8A-8C show a flowchart of a method of enabling low-vision users tointeract with touch-sensitive secondary displays, in accordance withsome embodiments.

FIGS. 9-11 illustrate functional block diagrams of a computing system,in accordance with some embodiments.

DESCRIPTION OF EMBODIMENTS

FIGS. 1A-1B, 2A-2D, 3A-3E, and 4 provide a description of exampledevices. FIGS. 5A-5N are schematics of a display used to illustrateexample user interfaces for enabling low-vision users to interact withtouch-sensitive secondary displays. FIGS. 6A-6C, 7A-7C, and 8A-8C areflowcharts of methods of enabling low-vision users to interact withtouch-sensitive secondary displays. The user interfaces in FIGS. 5A-5Nare used to illustrate the methods and/or processes in FIGS. 6A-6C,7A-7C, and 8A-8C.

Example Devices and Systems

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,it will be apparent to one of ordinary skill in the art that the variousdescribed embodiments may be practiced without these specific details.In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first contactcould be termed a second contact, and, similarly, a second contact couldbe termed a first contact, without departing from the scope of thevarious described embodiments. The first contact and the second contactare both contacts, but they are not the same contact.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a”, “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting,”depending on the context. Similarly, the phrase “if it is determined” or“if [a stated condition or event] is detected” is, optionally, construedto mean “upon determining” or “in response to determining” or “upondetecting [the stated condition or event]” or “in response to detecting[the stated condition or event],” depending on the context.

FIG. 1A is an illustrative diagram of a portable computing system 100,in accordance with some embodiments. Portable computing system 100 maybe, for example, a laptop computer, such as a MACBOOK® device, or anyother portable computing device. Portable computing system 100 includes:(A) a display portion 110 (also referred to herein as a first housing110 or housing 110) with a primary display 102; and (B) a body portion120 (also referred to as a second housing 120 or housing 120) with adynamic function row 104, a set of physical (i.e., movably actuated)keys 106, and a touchpad 108 partially contained within a same housing.Display portion 110 is typically mechanically, electrically, andcommunicatively coupled with body portion 120 of portable computingsystem 100. For example, portable computing system 100 may include ahinge, allowing display portion 110 to be rotated relative to bodyportion 120. Portable computing system 100 includes one or moreprocessors and memory storing one or more programs for execution by theone or more processors to perform any of the embodiments describedherein. In some embodiments, dynamic function row 104, which isdescribed in more detail with reference to FIG. 1B, is a touch screendisplay using resistive sensing, acoustic sensing, capacitive sensing,optical sensing, infrared sensing, or the like to detect user touchinputs and selections. In some embodiments, primary display 102 ofdisplay portion 110 is also a touch screen display.

FIG. 1B is an illustrative diagram of body portion 120 of portablecomputing system 100 in accordance with some embodiments. Body portion120 includes a set of physical keys 106 (also referred to herein as“physical keys 106” and “keyboard 106”), a dynamic function row 104, anda touchpad 108 partially contained within a same housing. In someembodiments, dynamic function row 104, which is a touch screen, replacesa function row of the set of physical keys 106 allowing the spaceconsumed by the set of physical keys 106 to be reduced, allowing for asmaller overall body portion 120 or allowing other portions, such astouchpad 108, to be larger. In some embodiments, dynamic function row104 is approximately 18 inches in length relative to a major dimensionof the set of physical keys 106. Although called a “row” for ease ofexplanation, in some other embodiments, the touch screen comprisingdynamic function row 104 in FIG. 1A may take any other form such as asquare, circle, a plurality of rows, column, a plurality of columns, aplurality of separate sectors, or the like. Although FIGS. 1A-1B showdynamic function row 104 replacing the function row of the set ofphysical keys 106, in some other embodiments, dynamic function row 104may additionally and/or alternatively replace a numpad section,editing/function section, or the like of the set of physical keys 106.

Each physical key of the set of physical keys 106 has at least oneassociated input. The input may be a printable character, non-printablecharacter, function, or other input. The input associated with aphysical key may be shown by a letter, word, symbol, or other indiciashown (e.g., printed) on the surface of the key in Latin script, Arabiccharacters, Chinese characters, or any other script. For example, theparticular physical key indicated at 138 is associated with alphabeticcharacter “z” as indicated by the letter z shown on the key. In anotherexample, a physical key labeled with the word “command” may beassociated with a command function. For example, the set of physicalkeys 106 is associated with a QWERTY, Dvorak, or other keyboard layoutswith alphanumeric, numeric, and/or editing/function sections (e.g.,standard, extended, or compact) according to ISO/IEC 9995, ANSI-INCITS154-1988, JIS X 6002-1980, or other similar standards.

A signal corresponding to an input associated with a physical key may bereceived by the processor of portable computing system 100 (or computingdevice 202 in FIGS. 2A-2D or peripheral keyboard 206 in FIGS. 2A-2B)when a key has been activated by a user. In an illustrative example,each key of the set of physical keys 106 includes two plates and aspring. A user may activate a key by pressing down on the key, whichcompresses the spring. When the spring is compressed, the two plates maycome into contact, allowing electric current to flow through theconnected plates. An input corresponding to the key may be provided to aprocessor in response to the flow of the current through the connectedplates. For example, in response to activation of one of the set of keys106 of peripheral keyboard 206 in FIG. 2C, an input corresponding to theactivated key is provided to computing device 202. It will be recognizedthat other systems for movably actuated keys could be used.

In some embodiments, dynamic function row 104 is a touch screen display(the dynamic function row is also referred to herein as atouch-sensitive secondary display 104) that displays one or moreuser-selectable symbols 142 (sometimes also herein called “userinterface elements,” “user interface components,” “affordances,”“buttons,” or “soft keys”). For example, dynamic function row 104replaces the function row keys on a typical keyboard. A user may selecta particular one of the one or more user-selectable symbols 142 bytouching a location on the touch screen display that corresponds to theparticular one of the one or more user-selectable symbols 142. Forexample, a user may select the user-selectable symbol indicated bymagnifying glass symbol 144 by tapping dynamic function row 104 suchthat the user's finger contacts dynamic function row 104 at the positionof the magnifying glass indicator 214. In some embodiments, a tapcontact or a tap gesture includes touch-down of a contact and lift-offof the contact within a predetermined amount of time (e.g., 250 ms orthe like). In some embodiments, the touch screen display of dynamicfunction row 104 is implemented using resistive sensing, acousticsensing, capacitive sensing, optical sensing, infrared sensing, or thelike to detect user inputs and selections.

When a user selects a particular one of the one or more user-selectablesymbols 142, a signal corresponding to the particular one of the one ormore user-selectable symbols 142 is generated by dynamic function row104. For example, when a user taps “esc” on dynamic function row 104,dynamic function row 104 transmits a signal indicating a user inputcorresponding to an escape function to the processor of portablecomputing system 100 (or computing device 202 in FIGS. 2A-2D).

In some embodiments, when a particular one of the one or moreuser-selectable symbols 142 is selected, dynamic function row 104transmits a signal corresponding to a position on the touch screendisplay where the particular one of the one or more user-selectablesymbols 142 is displayed, to the processor of portable computing system100 (or computing device 202 in FIGS. 2A-2D). For example, dynamicfunction row 104 may transmit a signal including a position value (0 to20) depending on the position on the touch screen display of theparticular one of the one or more user-selectable symbols 142 that wasselected. In the illustrative example of FIG. 1B, the “esc” symbol mayhave a position value of 0, magnifying glass symbol 144 may have aposition value of 16, and so on. A processor of portable computingsystem 100 (or computing device 202 in FIGS. 2A-2D) may receive thesignal indicating the position value of the selected user-selectablesymbol and interpret the position value using contextual information,such as an element of a graphical user interface displayed on primarydisplay 102 of display portion 110 (or peripheral display device 204,FIGS. 2A-2D) that is currently active or that has focus.

Each of the one or more user-selectable symbols 142 may include anindicator, such as a symbol (e.g., a magnifying glass symbol as shown at144), an abbreviated word (e.g., “esc”), an unabbreviated word, acharacter, an image, an animated image, a video, or the like. In someembodiments, a respective one of the one or more user-selectable symbols142 is capable of receiving user input(s).

An input may be associated with each of the one or more user-selectablesymbols 142. The input may be a function, character, numerical value,and the like. A respective one of the one or more user-selectablesymbols 142 may include an indicator that corresponds to the input forthe respective one of the one or more user-selectable symbols 142. Forexample, in FIG. 1B, the user-selectable symbol with the abbreviatedword “esc” indicates to the user that an escape function is associatedwith the user-selectable symbol. A function associated with the one ormore user-selectable symbols 142 may be activated when the user selectsa user-selectable symbol. For example, an escape function may beactivated when a user selects the user-selectable symbol with theindicator “esc.” Activation of the function may have different effectsdepending on the current state of portable computing system 100 (orcomputing device 202 in FIGS. 2A-2D). For example, when a dialog box isopen on primary display 102 of display portion 110 (or peripheraldisplay device 204, FIGS. 2A-2D), activating an escape function ondynamic function row 104 may close the dialog box. In another example,when a game application is being executed by a processor of portablecomputing system 100 (or computing device 202 in FIGS. 2A-2D),activating an escape function on dynamic function row 104 may pause thegame.

In some embodiments, functions may be associated with combinations ofmovably actuated keys and/or user-selectable symbols. For example,simultaneous actuation of a command key and “c” key (i.e., command+c)may be associated with a “copy” function. In another example,simultaneous actuation of the command key and selection of theuser-selectable symbol with the indicator “esc” (i.e., command+esc) mayactivate a function to open a particular application such as a mediaplayer application. In yet another example, simultaneous selection oftwo user-selectable symbols (e.g., the user-selectable symbol with theindicator “esc” and the user-selectable symbol 144 with the magnifyingglass indicator) may result in activation of a function, such as aspecialized search function.

In some embodiments, a first subset 146 of the one or moreuser-selectable symbols 142 of dynamic function row 104 may beassociated with one group of functions and a second subset 148 of theone or more user-selectable symbols 142 of dynamic function row 104 maybe associated with a second group of functions. For example, theuser-selectable symbols in first subset 146 may be global functions(e.g., system-level functions or affordances), and the user-selectablesymbols in second subset 148 may be application-specific functions. Assuch, the user-selectable symbols in second subset 148 change when thefocus shifts from a first element of a graphical user interfacedisplayed on primary display 102 (e.g., a first window corresponding toan Internet browser application) to a second element of the graphicaluser interface (e.g., a second window corresponding to an e-mailapplication). In contrast, the user-selectable symbols in first subset146 are maintained when the focus shifts from the first element of thegraphical user interface to the second element of the graphical userinterface.

In some embodiments, the user-selectable symbols in second subset 148are determined based on an active user interface element display onprimary display 102 that is in focus. In some embodiments, the term “infocus” can refer to the active element of the user interface (e.g., awindow associated with an application, a particular toolbar or menuassociated with an application, or the operating system) that iscurrently in the foreground and actively running or is controllable byinput received from a user of the computing system such as a key press,mouse click, voice command, gestural motion, or the like.

In some embodiments, the first subset 146 of the one or moreuser-selectable symbols 142 corresponding to global user-selectablesymbols occupies a first area of dynamic function row 104 (e.g., theleft half of dynamic function row 104), and the second subset 148 of theone or more user-selectable symbols 142 occupies a second area ofdynamic function row 104 (e.g., the right half of dynamic function row104). It will be realized that other proportions of dynamic function row104 may be allocated to the first subset 146 and the second subset 148.In some embodiments, when no application has focus, the second area ofdynamic function row 104 may not include any user-selectable symbols. Insome embodiments, dynamic function row 104 includes three or moresubsets of user-selectable symbols. In some embodiments, dynamicfunction row 104 includes a single set of user-selectable symbols thatare not divided into subsets. While a single row of user-selectablesymbols are shown in dynamic function row 104 in FIG. 1B, it will berecognized that dynamic function row 104 may include multiple rows ofuser-selectable symbols.

In some embodiments, the change in focus changes which element of thegraphical user interface displayed on primary display 102 of displayportion 110 (or peripheral display device 204, FIGS. 2A-2D) is activeand which element will receive user input. The user input may bereceived from a keyboard, mouse, touchpad, or other user input device.Additionally and/or alternatively, in some embodiments, the change infocus changes an element that is shown in the foreground of a graphicaluser interface displayed on primary display 102 of display portion 110(or peripheral display device 204, FIGS. 2A-2D).

In some embodiments, the change in focus occurs in response to userinput, for example, in response to user selection of an element of agraphical user interface (e.g., a different window) displayed on primarydisplay 102 of display portion 110 (or peripheral display device 204,FIGS. 2A-2D) or in response to user selection of a user-selectablesymbol (e.g., one of the affordances/symbols displayed on dynamicfunction row 104). The user selection may be a key stroke, a mouseclick, a mouse over, a command+tab input, or the like. In someembodiments, the change in focus occurs in response to a determinationby an operating system of portable system 100 (or computing device 202in FIGS. 2A-2D). For example, when a user closes an application windowthat has focus, the operating system may give focus to a differentapplication, such as an application that had focus prior to the closedapplication window. In another example, when a user closes anapplication window that has focus, the operating system may give focusto a dialog box prompting the user to save changes made to a documentvia the application.

In some embodiments, the change in focus may be a change from oneelement associated with an application to another element associatedwith the same application (e.g., from an e-mail composition window of ane-mail application to an inbox list window of an e-mail application orfrom one tab of an Internet browser application to another tab of anInternet browser application). In some embodiments, the change in focusmay be a change from an element associated with one application to anelement associated with another application (e.g., from an Internetbrowser window to an e-mail application window). Further, in someembodiments, the change in focus may be a change from an elementassociated with an application to an element associated with anoperating system, such as a system dialog box, a system setting control(e.g., volume control), a window associated with a file/foldernavigation application (e.g., Apple Inc.'s FINDER application), etc.Additionally, focus may also be directed to a dialog box, filedirectory, setting control (e.g., volume control), or any other elementof a graphical user interface for which information can be presented toa user and/or user input can be received.

FIG. 2A is an illustrative diagram of a first implementation of desktopcomputing system 200 in accordance with some embodiments. Desktopcomputing system 200 includes a computing device 202, a peripheraldisplay device 204 with primary display 102, a peripheral keyboard 206,and a peripheral mouse 208. Computing device 202 includes one or moreprocessors and memory storing one or more programs for execution by theone or more processors. In some embodiments, peripheral display device204 may be integrated with computing device 202 such as an iMAC® device.In some embodiments, primary display 102 of peripheral display device204 is a touch screen display. In FIG. 2A, peripheral display device 204(also referred to herein as a first housing 204 or housing 204),peripheral keyboard 206, and peripheral mouse 208 are communicativelycoupled to computing device 202 via a wired connection, such as USB orPS/2, or via a wireless communication link, using a communicationprotocol such as Bluetooth, Wi-Fi, or the like. For example, peripheralkeyboard 206 (also referred to herein as second housing 206 or housing206) is not more than fifteen feet from computing device 202 (e.g.approximately three feet away). In FIG. 2A, peripheral keyboard 206includes dynamic function row 104 and a set of physical keys 106 atleast partially contained within a same housing. In some embodiments,dynamic function row 104, which is described in more detail withreference to FIG. 1B, is a touch screen display. In some embodiments,peripheral keyboard 206 includes one or more processors and memorystoring one or more programs that may be executed by the one or moreprocessors of peripheral keyboard 206 to perform any of the embodimentsdescribed herein. In some embodiments, peripheral keyboard 206 relayssignals indicating user inputs (e.g., key strokes and selections ofuser-selectable symbols/affordances displayed by dynamic function row104) to computing device 202.

FIG. 2B is an illustrative diagram of a second implementation of desktopcomputing system 200 in accordance with some embodiments. In FIG. 2B,desktop computing system 200 includes a computing device 202, aperipheral display device 204 with primary display 102, and a peripheralkeyboard 206. In FIG. 2B, peripheral display device 204 and peripheralkeyboard 206 are communicatively coupled to computing device 202 via awired connection, such as USB or PS/2, or via a wireless communicationlink, using a communication protocol such as Bluetooth, Wi-Fi, or thelike. In FIG. 2B, peripheral keyboard 206 includes dynamic function row104, a set of physical keys 106, and touchpad 108 at least partiallycontained within a same housing. In some embodiments, dynamic functionrow 104, which is described in more detail with reference to FIG. 1B, isa touch screen display. In some embodiments, peripheral keyboard 206includes one or more processors and memory storing one or more programsthat may be executed by the one or more processors of peripheralkeyboard 206 to perform any of the embodiments described herein. In someembodiments, peripheral keyboard 206 relays signals indicating userinputs (e.g., key strokes, user interactions with touchpad 108, andselections of user-selectable symbols/affordances displayed by dynamicfunction row 104) to computing device 202.

FIG. 2C is an illustrative diagram of a third implementation of desktopcomputing system 200 in accordance with some embodiments. In FIG. 2C,desktop computing system 200 includes a computing device 202, aperipheral display device 204 with primary display 102, a peripheralkeyboard 206, and a first peripheral input mechanism 212. In FIG. 2C,peripheral display device 204, peripheral keyboard 206, and the firstperipheral input mechanism 212 are communicatively coupled to computingdevice 202 via a wired connection, such as USB or PS/2, or via awireless communication link, using a communication protocol such asBluetooth, Wi-Fi, or the like. In FIG. 2C, peripheral keyboard 206includes a set of physical keys 106, and the first peripheral inputmechanism 212 includes dynamic function row 104 and touchpad 108 atleast partially contained within a same housing. In some embodiments,dynamic function row 104, which is described in more detail withreference to FIG. 1B, is a touch screen display. In some embodiments,the first peripheral input mechanism 212 includes one or more processorsand memory storing one or more programs that may be executed by the oneor more processors of the first peripheral input mechanism 212 toperform any of the embodiments described herein. In some embodiments,the first peripheral input mechanism 212 relays signals indicating userinputs (e.g., user interactions with touchpad 108 and user selections ofuser-selectable symbols/affordances displayed by dynamic function row104) to computing device 202.

FIG. 2D is an illustrative diagram of a fourth implementation of desktopcomputing system 200 in accordance with some embodiments. In FIG. 2D,desktop computing system 200 includes a computing device 202, aperipheral display device 204 with primary display 102, a peripheralkeyboard 206, a peripheral mouse 208, and a second peripheral inputmechanism 222. In FIG. 2D, peripheral display device 204, peripheralkeyboard 206, peripheral mouse 208, and the second peripheral inputmechanism 222 are communicatively coupled to computing device 202 via awired connection, such as USB or PS/2, or via a wireless communicationlink, using a communication protocol such as Bluetooth, Wi-Fi, or thelike. In FIG. 2A, peripheral keyboard 206 includes dynamic function row104 and a set of physical keys 106. In FIG. 2D, peripheral keyboard 206includes a set of physical keys 106, and the second peripheral inputmechanism 222 includes dynamic function row 104 at least partiallycontained within the housing of the second peripheral input mechanism222. In some embodiments, dynamic function row 104, which is describedin more detail with reference to FIG. 1B, is a touch screen display. Insome embodiments, the second peripheral input mechanism 222 includes oneor more processors and memory storing one or more programs that may beexecuted by the one or more processors of the second peripheral inputmechanism 222 to perform any of the embodiments described herein. Insome embodiments, the second peripheral input mechanism 222 relayssignals indicating user inputs (e.g., user selections of user-selectablesymbols/affordances displayed by dynamic function row 104) to computingdevice 202.

FIG. 3A is a block diagram of an electronic device 300, in accordancewith some embodiments. In some embodiments, electronic device 300 is aportable electronic device, such as a laptop (e.g., portable computingsystem 100, FIG. 1A). In some embodiments, electronic device 300 is nota portable device, but is a desktop computer (e.g., computing device 202of desktop computing system 200, FIGS. 2A-2D), which is communicativelycoupled with a peripheral display system (e.g., peripheral displaydevice 204, FIGS. 2A-2D) and optionally a peripheral touch-sensitivesurface (e.g., a touchpad 108, FIGS. 2B-2C and/or a touch-sensitivedisplay, such as peripheral display device 204, FIGS. 2A-2D and/ordynamic function row 104, FIGS. 2A-2D).

Electronic device 300 typically supports a variety of applications, suchas one or more of the following: a drawing application, a presentationapplication, a word processing application, a website creationapplication, a disk authoring application, a spreadsheet application, agaming application, a video conferencing application, an e-mailapplication, an instant messaging application, an image managementapplication, a digital camera application, a digital video cameraapplication, a web browser application, and/or a media playerapplication.

The various applications that are executed on electronic device 300optionally use at least one common physical user-interface device, suchas the touch-sensitive surface. One or more functions of thetouch-sensitive surface as well as corresponding information displayedby electronic device 300 are, optionally, adjusted and/or varied fromone application to the next and/or within an application. In this way, acommon physical architecture (such as the touch-sensitive surface) ofelectronic device 300 optionally supports the variety of applicationswith user interfaces that are intuitive and transparent to the user.

Electronic device 300 includes memory 302 (which optionally includes oneor more computer readable storage mediums), memory controller 322, oneor more processing units (CPU(s)) 320, peripherals interface 318, RFcircuitry 308, audio circuitry 310, speaker 311, microphone 313,input/output (I/O) subsystem 306, other input or control devices 316,and external port 324. Electronic device 300 optionally includes adisplay system 312 (e.g., primary display 102 of display portion 110,FIG. 1A and/or dynamic function row 104, FIGS. 1A-1B), which may be atouch-sensitive display (sometimes also herein called a “touch screen”or a “touch screen display”). Electronic device 300 optionally includesone or more optical sensors 364. Electronic device 300 optionallyincludes one or more intensity sensors 365 for detecting intensity ofcontacts on a touch-sensitive surface such as touch-sensitive display ora touchpad. Electronic device 300 optionally includes one or moretactile output generators 367 for generating tactile outputs on atouch-sensitive surface such as touch-sensitive display or a touchpad(e.g., touchpad 108, FIGS. 1A-1B). These components optionallycommunicate over one or more communication buses or signal lines 303.

As used in the specification, the term “intensity” of a contact on atouch-sensitive surface refers to the force or pressure (force per unitarea) of a contact (e.g., a finger contact) on the touch sensitivesurface, or to a substitute (proxy) for the force or pressure of acontact on the touch sensitive surface. The intensity of a contact has arange of values that includes at least four distinct values and moretypically includes hundreds of distinct values (e.g., at least 256).Intensity of a contact is, optionally, determined (or measured) usingvarious approaches and various sensors or combinations of sensors. Forexample, one or more force sensors underneath or adjacent to thetouch-sensitive surface are, optionally, used to measure force atvarious points on the touch-sensitive surface. In some implementations,force measurements from multiple force sensors are combined (e.g., aweighted average) to determine an estimated force of a contact.Similarly, a pressure-sensitive tip of a stylus is, optionally, used todetermine a pressure of the stylus on the touch-sensitive surface.Alternatively, the size of the contact area detected on thetouch-sensitive surface and/or changes thereto, the capacitance of thetouch-sensitive surface proximate to the contact and/or changes thereto,and/or the resistance of the touch-sensitive surface proximate to thecontact and/or changes thereto are, optionally, used as a substitute forthe force or pressure of the contact on the touch-sensitive surface. Insome implementations, the substitute measurements for contact force orpressure are used directly to determine whether an intensity thresholdhas been exceeded (e.g., the intensity threshold is described in unitscorresponding to the substitute measurements). In some implementations,the substitute measurements for contact force or pressure are convertedto an estimated force or pressure and the estimated force or pressure isused to determine whether an intensity threshold has been exceeded(e.g., the intensity threshold is a pressure threshold measured in unitsof pressure).

As used in the specification and claims, the term “tactile output”refers to physical displacement of a device relative to a previousposition of the device, physical displacement of a component (e.g., atouch-sensitive surface) of a device relative to another component(e.g., housing) of the device, or displacement of the component relativeto a center of mass of the device that will be detected by a user withthe user's sense of touch. For example, in situations where the deviceor the component of the device is in contact with a surface of a userthat is sensitive to touch (e.g., a finger, palm, or other part of auser's hand), the tactile output generated by the physical displacementwill be interpreted by the user as a tactile sensation corresponding toa perceived change in physical characteristics of the device or thecomponent of the device. For example, movement of a touch-sensitivesurface (e.g., a touch-sensitive display or touch/track pad) is,optionally, interpreted by the user as a “down click” or “up click” of aphysical actuator button. In some cases, a user will feel a tactilesensation such as an “down click” or “up click” even when there is nomovement of a physical actuator button associated with thetouch-sensitive surface that is physically pressed (e.g., displaced) bythe user's movements. As another example, movement of thetouch-sensitive surface is, optionally, interpreted or sensed by theuser as “roughness” of the touch-sensitive surface, even when there isno change in smoothness of the touch-sensitive surface. While suchinterpretations of touch by a user will be subject to the individualizedsensory perceptions of the user, there are many sensory perceptions oftouch that are common to a large majority of users. Thus, when a tactileoutput is described as corresponding to a particular sensory perceptionof a user (e.g., an “up click,” a “down click,” “roughness”), unlessotherwise stated, the generated tactile output corresponds to physicaldisplacement of the device or a component thereof that will generate thedescribed sensory perception for a typical (or average) user.

It should be appreciated that electronic device 300 is only an exampleand that electronic device 300 optionally has more or fewer componentsthan shown, optionally combines two or more components, or optionallyhas a different configuration or arrangement of the components. Thevarious components shown in FIG. 3A are implemented in hardware,software, firmware, or a combination thereof, including one or moresignal processing and/or application specific integrated circuits.

Memory 302 optionally includes high-speed random access memory andoptionally also includes non-volatile memory, such as one or moremagnetic disk storage devices, flash memory devices, or othernon-volatile solid-state memory devices. Access to memory 302 by othercomponents of electronic device 300, such as CPU(s) 320 and peripheralsinterface 318, is, optionally, controlled by memory controller 322.Peripherals interface 318 can be used to couple input and outputperipherals to CPU(s) 320 and memory 302. The one or more processingunits 320 run or execute various software programs and/or sets ofinstructions stored in memory 302 to perform various functions forelectronic device 300 and to process data. In some embodiments,peripherals interface 318, CPU(s) 320, and memory controller 322 are,optionally, implemented on a single chip, such as chip 304. In someother embodiments, they are, optionally, implemented on separate chips.

RF (radio frequency) circuitry 308 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 308 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 308 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 308 optionally communicates with networks, such as theInternet, also referred to as the World Wide Web (WWW), an intranetand/or a wireless network, such as a cellular telephone network, awireless local area network (LAN) and/or a metropolitan area network(MAN), and other devices by wireless communication. The wirelesscommunication optionally uses any of a plurality of communicationsstandards, protocols and technologies, including but not limited toGlobal System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), high-speed downlink packet access (HSDPA),high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO),HSPA, HSPA+, Dual-Cell HSPA (DC-HSPDA), long term evolution (LTE), nearfield communication (NFC), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a,IEEE 802.11b, IEEE 802.11g, and/or IEEE 802.11n), voice over InternetProtocol (VoIP), Wi-MAX, a protocol for e-mail (e.g., Internet messageaccess protocol (IMAP) and/or post office protocol (POP)), instantmessaging (e.g., extensible messaging and presence protocol (XMPP),Session Initiation Protocol for Instant Messaging and PresenceLeveraging Extensions (SIMPLE), Instant Messaging and Presence Service(IMPS)), and/or Short Message Service (SMS), or any other suitablecommunication protocol, including communication protocols not yetdeveloped as of the filing date of this document.

Audio circuitry 310, speaker 311, and microphone 313 provide an audiointerface between a user and electronic device 300. Audio circuitry 310receives audio data from peripherals interface 318, converts the audiodata to an electrical signal, and transmits the electrical signal tospeaker 311. Speaker 311 converts the electrical signal to human-audiblesound waves. Audio circuitry 310 also receives electrical signalsconverted by microphone 313 from sound waves. Audio circuitry 310converts the electrical signals to audio data and transmits the audiodata to peripherals interface 318 for processing. Audio data is,optionally, retrieved from and/or transmitted to memory 302 and/or RFcircuitry 308 by peripherals interface 318. In some embodiments, audiocircuitry 310 also includes a headset jack. The headset jack provides aninterface between audio circuitry 310 and removable audio input/outputperipherals, such as output-only headphones or a headset with bothoutput (e.g., a headphone for one or both ears) and input (e.g., amicrophone).

I/O subsystem 306 couples the input/output peripherals of electronicdevice 300, such as display system 312 and other input or controldevices 316, to peripherals interface 318. I/O subsystem 306 optionallyincludes display controller 356, optical sensor controller 358,intensity sensor controller 359, haptic feedback controller 361, and oneor more other input controllers 360 for other input or control devices.The one or more other input controllers 360 receive/send electricalsignals from/to other input or control devices 316. The other input orcontrol devices 316 optionally include physical buttons (e.g., pushbuttons, rocker buttons, etc.), dials, slider switches, joysticks, clickwheels, and so forth. In some alternate embodiments, other inputcontroller(s) 360 are, optionally, coupled with any (or none) of thefollowing: a keyboard, infrared port, USB port, and a pointer devicesuch as a mouse. The one or more physical buttons optionally include anup/down button for volume control of speaker 311 and/or microphone 313.

Display system 312 (e.g., primary display 102 of display portion 110,FIG. 1A and/or dynamic function row 104, FIGS. 1A-1B) provides an outputinterface (and, optionally, an input interface when it is atouch-sensitive display) between electronic device 300 and a user.Display controller 356 receives and/or sends electrical signals from/todisplay system 312. Display system 312 displays visual output to theuser. The visual output optionally includes graphics, text, icons,video, and any combination thereof (collectively termed “graphics”). Insome embodiments, some or all of the visual output corresponds touser-interface objects/elements.

In some embodiments, display system 312 (e.g., primary display 102 ofdisplay portion 110, FIG. 1A and/or dynamic function row 104, FIGS.1A-1B) is a touch-sensitive display with a touch-sensitive surface,sensor, or set of sensors that accepts input from the user based onhaptic and/or tactile contact. As such, display system 312 and displaycontroller 356 (along with any associated modules and/or sets ofinstructions in memory 302) detect contact (and any movement or breakingof the contact) on display system 312 and convert the detected contactinto interaction with user-interface objects (e.g., one or more softkeys, icons, web pages, or images) that are displayed on display system312. In one example embodiment, a point of contact between displaysystem 312 and the user corresponds to an area under a finger of theuser.

Display system 312 (e.g., primary display 102 of display portion 110,FIG. 1A and/or dynamic function row 104, FIGS. 1A-1B) optionally usesLCD (liquid crystal display) technology, LPD (light emitting polymerdisplay) technology, LED (light emitting diode) technology, or OLED(organic light emitting diode) technology, although other displaytechnologies are used in other embodiments. In some embodiments, whendisplay system 312 is a touch-sensitive display, display system 312 anddisplay controller 356 optionally detect contact and any movement orbreaking thereof using any of a plurality of touch sensing technologiesnow known or later developed, including but not limited to capacitive,resistive, infrared, and surface acoustic wave technologies, as well asother proximity sensor arrays or other elements for determining one ormore points of contact with display system 312. In one exampleembodiment, projected mutual capacitance sensing technology is used,such as that found in the iPHONE®, iPODTOUCH®, and iPAD® from Apple Inc.of Cupertino, Calif.

Display system 312 (e.g., primary display 102 of display portion 110,FIG. 1A and/or dynamic function row 104, FIGS. 1A-1B) optionally has avideo resolution in excess of 400 dpi (e.g., 500 dpi, 800 dpi, orgreater). In some embodiments, display system 312 is a touch-sensitivedisplay with which the user optionally makes contact using a stylus, afinger, and so forth. In some embodiments, the user interface isdesigned to work primarily with finger-based contacts and gestures. Insome embodiments, electronic device 300 translates the roughfinger-based input into a precise pointer/cursor position or command forperforming the actions desired by the user.

In some embodiments, in addition to display system 312, electronicdevice 300 optionally includes a touchpad (e.g., touchpad 108, FIGS.1A-1B) for activating or deactivating particular functions. In someembodiments, the touchpad is a touch-sensitive area of electronic device300 that, unlike display system 312, does not display visual output. Insome embodiments, when display system 312 is a touch-sensitive display,the touchpad is, optionally, a touch-sensitive surface that is separatefrom display system 312, or an extension of the touch-sensitive surfaceformed by display system 312.

Electronic device 300 also includes power system 362 for powering thevarious components. Power system 362 optionally includes a powermanagement system, one or more power sources (e.g., battery, alternatingcurrent (AC), etc.), a recharging system, a power failure detectioncircuit, a power converter or inverter, a power status indicator (e.g.,a light-emitting diode (LED)) and any other components associated withthe generation, management and distribution of power in portabledevices.

Electronic device 300 optionally also includes one or more opticalsensors 364 coupled with optical sensor controller 358 in I/O subsystem306. Optical sensor(s) 364 optionally includes charge-coupled device(CCD) or complementary metal-oxide semiconductor (CMOS)phototransistors. Optical sensor(s) 364 receive light from theenvironment, projected through one or more lens, and converts the lightto data representing an image. In conjunction with imaging module 343,optical sensor(s) 364 optionally capture still images or video. In someembodiments, an optical sensor is located on the front of electronicdevice 300 so that the user's image is, optionally, obtained forvideoconferencing while the user views the other video conferenceparticipants on display system 312.

Electronic device 300 optionally also includes one or more contactintensity sensor(s) 365 coupled with intensity sensor controller 359 inI/O subsystem 306. Contact intensity sensor(s) 365 optionally includesone or more piezoresistive strain gauges, capacitive force sensors,electric force sensors, piezoelectric force sensors, optical forcesensors, capacitive touch-sensitive surfaces, or other intensity sensors(e.g., sensors used to measure the force (or pressure) of a contact on atouch-sensitive surface). Contact intensity sensor(s) 365 receivescontact intensity information (e.g., pressure information or a proxy forpressure information) from the environment. In some embodiments, atleast one contact intensity sensor is collocated with, or proximate to,a touch-sensitive surface (e.g., touchpad 108, FIGS. 1A-1B or displaysystem 312 when it is a touch-sensitive display).

Electronic device 300 optionally also includes one or more tactileoutput generators 367 coupled with haptic feedback controller 361 in I/Osubsystem 306. Tactile output generator(s) 367 optionally includes oneor more electroacoustic devices such as speakers or other audiocomponents and/or electromechanical devices that convert energy intolinear motion such as a motor, solenoid, electroactive polymer,piezoelectric actuator, electrostatic actuator, or other tactile outputgenerating component (e.g., a component that converts electrical signalsinto tactile outputs on the device). Contact intensity sensor(s) 365receives tactile feedback generation instructions from haptic feedbackmodule 333 and generates tactile outputs that are capable of beingsensed by a user of electronic device 300. In some embodiments, at leastone tactile output generator is collocated with, or proximate to, atouch-sensitive surface (e.g., touchpad 108, FIGS. 1A-1B or displaysystem 312 when it is a touch-sensitive display) and, optionally,generates a tactile output by moving the touch-sensitive surfacevertically (e.g., in/out of a surface of electronic device 300) orlaterally (e.g., back and forth in the same plane as a surface ofelectronic device 300).

Electronic device 300 optionally also includes one or more proximitysensors 366 coupled with peripherals interface 318. Alternately,proximity sensor(s) 366 are coupled with other input controller(s) 360in I/O subsystem 306. Electronic device 300 optionally also includes oneor more accelerometers 368 coupled with peripherals interface 318.Alternately, accelerometer(s) 368 are coupled with other inputcontroller(s) 360 in I/O subsystem 306.

In some embodiments, the software components stored in memory 302include operating system 326, communication module 328 (or set ofinstructions), contact/motion module 330 (or set of instructions),graphics module 332 (or set of instructions), applications 340 (or setsof instructions), and dynamic function row module 350 (or sets ofinstructions). Furthermore, in some embodiments, memory 302 storesdevice/global internal state 357 (or sets of instructions), as shown inFIG. 3A. Device/global internal state 357 includes one or more of:active application state, indicating which applications, if any, arecurrently active and/or in focus; display state, indicating whatapplications, views or other information occupy various regions ofdisplay system 312 (e.g., primary display 102 of display portion 110,FIG. 1A and/or dynamic function row 104, FIGS. 1A-1B) and/or aperipheral display system (e.g., primary display 102 of peripheraldisplay device 204, FIGS. 2A-2D and/or dynamic function row 104, FIGS.2A-2D); sensor state, including information obtained from varioussensors and input or control devices 316 of electronic device 300; andlocation information concerning the location and/or attitude ofelectronic device 300.

Operating system 326 (e.g., DARWIN, RTXC, LINUX, UNIX, OS X, WINDOWS, oran embedded operating system such as VXWorks) includes various softwarecomponents and/or drivers for controlling and managing general systemtasks (e.g., memory management, storage device control, powermanagement, etc.) and facilitates communication between various hardwareand software components.

Communication module 328 facilitates communication with other devices(e.g., computing device 202, FIGS. 2A-2D; peripheral mouse 208, FIGS. 2Aand 2D; peripheral keyboard 206, FIGS. 2A-2B; first peripheral inputmechanism 212, FIG. 2C; and/or second peripheral input mechanism 222,FIG. 2D) over one or more external ports 324 and/or RF circuitry 308 andalso includes various software components for sending/receiving data viaRF circuitry 308 and/or external port 324. External port 324 (e.g.,Universal Serial Bus (USB), FIREWIRE, etc.) is adapted for couplingdirectly to other devices or indirectly over a network (e.g., theInternet, wireless LAN, etc.). In some embodiments, external port 324 isa multi-pin (e.g., 30-pin) connector that is the same as, or similar toand/or compatible with the 30-pin connector used on iPod® devices.

Contact/motion module 330 optionally detects contact with display system312 when it is a touch-sensitive display (in conjunction with displaycontroller 356) and other touch sensitive devices (e.g., a touchpad orphysical click wheel). Contact/motion module 330 includes varioussoftware components for performing various operations related todetection of contact, such as determining if contact has occurred (e.g.,detecting a finger-down event), determining an intensity of the contact(e.g., the force or pressure of the contact or a substitute for theforce or pressure of the contact), determining if there is movement ofthe contact and tracking the movement across the touch-sensitive surface(e.g., detecting one or more finger-dragging events), and determining ifthe contact has ceased (e.g., detecting a finger-up event or a break incontact). Contact/motion module 330 receives contact data from thetouch-sensitive surface. Determining movement of the point of contact,which is represented by a series of contact data, optionally includesdetermining speed (magnitude), velocity (magnitude and direction),and/or an acceleration (a change in magnitude and/or direction) of thepoint of contact. These operations are, optionally, applied to singlecontacts (e.g., one finger contacts) or to multiple simultaneouscontacts (e.g., “multitouch”/multiple finger contacts). In someembodiments, contact/motion module 330 also detects contact on atouchpad (e.g., touchpad 108, FIGS. 1A-1B).

In some embodiments, contact/motion module 330 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has selected or“clicked” on an affordance). In some embodiments at least a subset ofthe intensity thresholds are determined in accordance with softwareparameters (e.g., the intensity thresholds are not determined by theactivation thresholds of particular physical actuators and can beadjusted without changing the physical hardware of electronic device300). For example, a mouse “click” threshold of a trackpad or touchscreen display can be set to any of a large range of predefinedthresholds values without changing the trackpad or touch screen displayhardware. Additionally, in some implementations a user of the device isprovided with software settings for adjusting one or more of the set ofintensity thresholds (e.g., by adjusting individual intensity thresholdsand/or by adjusting a plurality of intensity thresholds at once with asystem-level click “intensity” parameter).

Contact/motion module 330 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns (e.g., different motions, timings, and/or intensities ofdetected contacts). Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap contact includes detecting a finger-down event followed by detectinga finger-up (lift off) event at the same position (or substantially thesame position) as the finger-down event (e.g., at the position of anicon). As another example, detecting a finger swipe gesture on thetouch-sensitive surface includes detecting a finger-down event followedby detecting one or more finger-dragging events, and in some embodimentsalso followed by detecting a finger-up (lift off) event.

Graphics module 332 includes various known software components forrendering and causing display of graphics on primary display 102 (e.g.,primary display 102 of display portion 110, FIG. 1A or primary display102 of peripheral display device 204, FIGS. 2A-2D) or other display,including components for changing the visual impact (e.g., brightness,transparency, saturation, contrast or other visual property) of graphicsthat are displayed. As used herein, the term “graphics” includes anyobject that can be displayed to a user, including without limitationtext, web pages, icons (such as user-interface objects including softkeys), digital images, videos, animations and the like. In someembodiments, graphics module 332 stores data representing graphics to beused. Each graphic is, optionally, assigned a corresponding code.Graphics module 332 receives, from applications etc., one or more codesspecifying graphics to be displayed along with, if necessary, coordinatedata and other graphic property data, and then generates screen imagedata to output to display controller 356.

Haptic feedback module 333 includes various software components forgenerating instructions used by tactile output generator(s) 367 toproduce tactile outputs at one or more locations on electronic device300 in response to user interactions with electronic device 300.

Applications 340 optionally include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   -   e-mail client module 341 (sometimes also herein called “mail        app” or “e-mail app”) for receiving, sending, composing, and        viewing e-mails;    -   imaging module 342 for capturing still and/or video images;    -   image management module 343 (sometimes also herein called “photo        app”) for editing and viewing still and/or video images;    -   media player module 344 (sometimes also herein called “media        player app”) for playback of audio and/or video; and    -   web browsing module 345 (sometimes also herein called “web        browser”) for connecting to and browsing the Internet.

Examples of other applications 340 that are, optionally, stored inmemory 302 include messaging and communications applications, wordprocessing applications, other image editing applications, drawingapplications, presentation applications, JAVA-enabled applications,encryption applications, digital rights management applications, voicerecognition applications, and voice replication applications.

In conjunction with one or more of RF circuitry 308, display system 312(e.g., primary display 102 of display portion 110, FIG. 1A and/ordynamic function row 104, FIGS. 1A-1B), display controller 356, andcontact module 330, graphics module 332, e-mail client module 341includes executable instructions to create, send, receive, and managee-mail in response to user instructions. In conjunction with imagemanagement module 343, e-mail client module 341 makes it very easy tocreate and send e-mails with still or video images taken with imagingmodule 342.

In conjunction with one or more of display system 312 (e.g., primarydisplay 102 of display portion 110, FIG. 1A and/or dynamic function row104, FIGS. 1A-1B), display controller 356, optical sensor(s) 364,optical sensor controller 358, contact module 330, graphics module 332,and image management module 343, imaging module 342 includes executableinstructions to capture still images or video (including a video stream)and store them into memory 302, modify characteristics of a still imageor video, or delete a still image or video from memory 302.

In conjunction with one or more of display system 312 (e.g., primarydisplay 102 of display portion 110, FIG. 1A and/or dynamic function row104, FIGS. 1A-1B), display controller 356, contact module 330, graphicsmodule 332, and imaging module 342, image management module 343 includesexecutable instructions to arrange, modify (e.g., edit), or otherwisemanipulate, label, delete, present (e.g., in a digital slide show oralbum), and store still and/or video images.

In conjunction with one or more of display system 312 (e.g., primarydisplay 102 of display portion 110, FIG. 1A and/or dynamic function row104, FIGS. 1A-1B), display controller 356, contact module 330, graphicsmodule 332, audio circuitry 310, speaker 311, RF circuitry 308, and webbrowsing module 345, media player module 344 includes executableinstructions that allow the user to download and play back recordedmusic and other sound files stored in one or more file formats, such asMP3 or AAC files, and executable instructions to display, present orotherwise play back videos (e.g., on primary display 102 of displayportion 110, FIG. 1A or primary display 102 of peripheral display device2014, FIGS. 2A-2B connected via external port 324).

In conjunction with one or more of RF circuitry 308, display system 312(e.g., primary display 102 of display portion 110, FIG. 1A and/ordynamic function row 104, FIGS. 1A-1B), display controller 356, contactmodule 330, and graphics module 332, web browsing module 345 includesexecutable instructions to browse the Internet in accordance with userinstructions, including searching, linking to, receiving, and displayingweb pages or portions thereof, as well as attachments and other fileslinked to web pages.

Dynamic function row (DFR) module 350 includes: focus determining module351, DFR determining module 352, and DFR presenting module 353. In someembodiments, focus determining module 351 is configured to determine anactive user interface element that is in focus on the graphical userinterface displayed by display system 312 (e.g., primary display 102 ofdisplay portion 110, FIG. 1A) or a peripheral display system (e.g.,peripheral display device 204, FIGS. 2A-2D). In some embodiments, DFRdetermining module 352 is configured to determine graphics (e.g., a setof one or more affordances) based on the active user interface elementthat is in focus. In some embodiments, DFR presenting module 353 isconfigured to render the graphics determined by DFR determining module352 on display system 312 (e.g., dynamic function row 104, FIGS. 1A-1B).DFR presenting module 353 includes various known software components forrendering and causing display of graphics on display system 312 (e.g.,dynamic function row 104, FIGS. 1A-1B), including components forchanging the visual impact (e.g., brightness, transparency, saturation,contrast or other visual property) of graphics that are displayed. Asused herein, the term “graphics” includes any object that can bedisplayed to a user, including without limitation text, web pages, icons(such as user-interface objects including soft keys), digital images,videos, animations, and the like. In some embodiments, DFR module 350includes other modules for: adjusting the sensitivity of dynamicfunction row 104; adjusting the audible and/or haptic feedback providedby dynamic function row 104; adjusting the settings of affordances andinformation displayed by dynamic function row 104 (e.g., size,brightness, font, language, and the like); adjusting the current powermode of dynamic function row 104 (e.g., normal and low-power modes); andthe like.

In some embodiments, the dynamic function row module 350 interfaces withcomponents that allow for providing predicted/proactive/suggestedcontent items (including predicted recipients, suggested text completionstrings, proactively suggested applications, etc.). Proactivelysuggesting content items is discussed in more detail in U.S. applicationSer. No. 15/167,713, which is hereby incorporated by reference in itsentirety.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (i.e., sets of instructions) need notbe implemented as separate software programs, procedures or modules, andthus various subsets of these modules are, optionally, combined orotherwise re-arranged in various embodiments. In some embodiments,memory 302 optionally stores a subset of the modules and data structuresidentified above. Furthermore, memory 302 optionally stores additionalmodules and data structures not described above.

FIG. 3B is a block diagram of components for event handling of FIG. 3A,in accordance with some embodiments. In some embodiments, memory 302(FIG. 3A) includes event sorter 370 (e.g., in operating system 326) andan application 340-1 (e.g., any of the aforementioned applications 341,342, 343, 344, or 345).

Event sorter 370 receives event information and determines theapplication 340-1 and application view 391 of application 340-1 to whichto deliver the event information. Event sorter 370 includes eventmonitor 371 and event dispatcher module 374. In some embodiments,application 340-1 includes application internal state 392, whichindicates the current application view(s) displayed on display system312 (e.g., primary display 102 of display portion 110, FIG. 1A and/ordynamic function row 104, FIGS. 1A-1B) when the application is active orexecuting. In some embodiments, device/global internal state 357 is usedby event sorter 370 to determine which application(s) is (are) currentlyactive or in focus, and application internal state 392 is used by eventsorter 370 to determine application views 391 to which to deliver eventinformation.

In some embodiments, application internal state 392 includes additionalinformation, such as one or more of: resume information to be used whenapplication 340-1 resumes execution, user interface state informationthat indicates information being displayed or that is ready for displayby application 340-1, a state queue for enabling the user to go back toa prior state or view of application 340-1, and a redo/undo queue ofprevious actions taken by the user.

Event monitor 371 receives event information from peripherals interface318. Event information includes information about a sub-event (e.g., auser touch on display system 312 when it is a touch-sensitive display,as part of a multi-touch gesture). Peripherals interface 318 transmitsinformation it receives from I/O subsystem 306 or a sensor, such asproximity sensor(s) 366, accelerometer(s) 368, and/or microphone 313(through audio circuitry 310). Information that peripherals interface318 receives from I/O subsystem 306 includes information from displaysystem 312 when it is a touch-sensitive display or anothertouch-sensitive surface (e.g., touchpad 108, FIGS. 1A-1B).

In some embodiments, event monitor 371 sends requests to the peripheralsinterface 318 at predetermined intervals. In response, peripheralsinterface 318 transmits event information. In other embodiments,peripheral interface 318 transmits event information only when there isa significant event (e.g., receiving an input above a predeterminednoise threshold and/or for more than a predetermined duration).

In some embodiments, event sorter 370 also includes a hit viewdetermination module 372 and/or an active event recognizer determinationmodule 373.

Hit view determination module 372 provides software procedures fordetermining where a sub-event has taken place within one or more views,when display system 312 displays more than one view, where views aremade up of controls and other elements that a user can see on thedisplay.

Another aspect of the user interface associated with an application is aset of views, sometimes herein called application views or userinterface windows, in which information is displayed and touch-basedgestures occur. The application views (of an application) in which atouch is detected optionally correspond to programmatic levels within aprogrammatic or view hierarchy of the application. For example, thelowest level view in which a touch is detected is, optionally, calledthe hit view, and the set of events that are recognized as proper inputsare, optionally, determined based, at least in part, on the hit view ofthe initial touch that begins a touch-based gesture.

Hit view determination module 372 receives information related tosub-events of a touch-based gesture. When an application has multipleviews organized in a hierarchy, hit view determination module 372identifies a hit view as the lowest view in the hierarchy which shouldhandle the sub-event. In most circumstances, the hit view is the lowestlevel view in which an initiating sub-event occurs (i.e., the firstsub-event in the sequence of sub-events that form an event or potentialevent). Once the hit view is identified by the hit view determinationmodule, the hit view typically receives all sub-events related to thesame touch or input source for which it was identified as the hit view.

Active event recognizer determination module 373 determines which viewor views within a view hierarchy should receive a particular sequence ofsub-events. In some embodiments, active event recognizer determinationmodule 373 determines that only the hit view should receive a particularsequence of sub-events. In other embodiments, active event recognizerdetermination module 373 determines that all views that include thephysical location of a sub-event are actively involved views, andtherefore determines that all actively involved views should receive aparticular sequence of sub-events. In other embodiments, even if touchsub-events were entirely confined to the area associated with oneparticular view, views higher in the hierarchy would still remain asactively involved views.

Event dispatcher module 374 dispatches the event information to an eventrecognizer (e.g., event recognizer 380). In embodiments including activeevent recognizer determination module 373, event dispatcher module 374delivers the event information to an event recognizer determined byactive event recognizer determination module 373. In some embodiments,event dispatcher module 374 stores in an event queue the eventinformation, which is retrieved by a respective event receiver 382.

In some embodiments, operating system 326 includes event sorter 370.Alternatively, application 340-1 includes event sorter 370. In yet otherembodiments, event sorter 370 is a stand-alone module, or a part ofanother module stored in memory 302, such as contact/motion module 330.

In some embodiments, application 340-1 includes a plurality of eventhandlers 390 and one or more application views 391, each of whichincludes instructions for handling touch events that occur within arespective view of the application's user interface. Each applicationview 391 of the application 340-1 includes one or more event recognizers380. Typically, an application view 391 includes a plurality of eventrecognizers 380. In other embodiments, one or more of event recognizers380 are part of a separate module, such as a user interface kit (notshown) or a higher level object from which application 340-1 inheritsmethods and other properties. In some embodiments, a respective eventhandler 390 includes one or more of: data updater 376, object updater377, GUI updater 378, and/or event data 379 received from event sorter370. Event handler 390 optionally utilizes or calls data updater 376,object updater 377 or GUI updater 378 to update the application internalstate 392. Alternatively, one or more of the application views 391includes one or more respective event handlers 390. Also, in someembodiments, one or more of data updater 376, object updater 377, andGUI updater 378 are included in an application view 391.

A respective event recognizer 380 receives event information (e.g.,event data 379) from event sorter 370, and identifies an event from theevent information. Event recognizer 380 includes event receiver 382 andevent comparator 384. In some embodiments, event recognizer 380 alsoincludes at least a subset of: metadata 383, and event deliveryinstructions 388 (which optionally include sub-event deliveryinstructions).

Event receiver 382 receives event information from event sorter 370. Theevent information includes information about a sub-event, for example, atouch or a touch movement. Depending on the sub-event, the eventinformation also includes additional information, such as location ofthe sub-event. When the sub-event concerns motion of a touch, the eventinformation optionally also includes speed and direction of thesub-event. In some embodiments, events include rotation of the devicefrom one orientation to another (e.g., from a portrait orientation to alandscape orientation, or vice versa), and the event informationincludes corresponding information about the current orientation (alsocalled device attitude) of the device.

Event comparator 384 compares the event information to predefined eventor sub-event definitions and, based on the comparison, determines anevent or sub-event, or determines or updates the state of an event orsub-event. In some embodiments, event comparator 384 includes eventdefinitions 386. Event definitions 386 contain definitions of events(e.g., predefined sequences of sub-events), for example, event 1(387-1), event 2 (387-2), and others. In some embodiments, sub-events inan event 387 include, for example, touch begin, touch end, touchmovement, touch cancellation, and multiple touching. In one example, thedefinition for event 1 (387-1) is a double tap on a displayed object.The double tap, for example, comprises a first touch (touch begin) onthe displayed object for a predetermined phase, a first lift-off (touchend) for a predetermined phase, a second touch (touch begin) on thedisplayed object for a predetermined phase, and a second lift-off (touchend) for a predetermined phase. In another example, the definition forevent 2 (387-2) is a dragging on a displayed object. The dragging, forexample, comprises a touch (or contact) on the displayed object for apredetermined phase, a movement of the touch across display system 312when it is a touch-sensitive display, and lift-off of the touch (touchend). In some embodiments, the event also includes information for oneor more associated event handlers 390.

In some embodiments, event definition 387 includes a definition of anevent for a respective user-interface object. In some embodiments, eventcomparator 384 performs a hit test to determine which user-interfaceobject is associated with a sub-event. For example, in an applicationview in which three user-interface objects are displayed on displaysystem 312, when a touch is detected on display system 312 when it is atouch-sensitive display, event comparator 384 performs a hit test todetermine which of the three user-interface objects is associated withthe touch (sub-event). If each displayed object is associated with arespective event handler 390, the event comparator uses the result ofthe hit test to determine which event handler 390 should be activated.For example, event comparator 384 selects an event handler associatedwith the sub-event and the object triggering the hit test.

In some embodiments, the definition for a respective event 387 alsoincludes delayed actions that delay delivery of the event informationuntil after it has been determined whether the sequence of sub-eventsdoes or does not correspond to the event recognizer's event type.

When a respective event recognizer 380 determines that the series ofsub-events do not match any of the events in event definitions 386, therespective event recognizer 380 enters an event impossible, eventfailed, or event ended state, after which it disregards subsequentsub-events of the touch-based gesture. In this situation, other eventrecognizers, if any, that remain active for the hit view continue totrack and process sub-events of an ongoing touch-based gesture.

In some embodiments, a respective event recognizer 380 includes metadata383 with configurable properties, flags, and/or lists that indicate howthe event delivery system should perform sub-event delivery to activelyinvolved event recognizers. In some embodiments, metadata 383 includesconfigurable properties, flags, and/or lists that indicate how eventrecognizers interact, or are enabled to interact, with one another. Insome embodiments, metadata 383 includes configurable properties, flags,and/or lists that indicate whether sub-events are delivered to varyinglevels in the view or programmatic hierarchy.

In some embodiments, a respective event recognizer 380 activates eventhandler 390 associated with an event when one or more particularsub-events of an event are recognized. In some embodiments, a respectiveevent recognizer 380 delivers event information associated with theevent to event handler 390. Activating an event handler 390 is distinctfrom sending (and deferred sending) sub-events to a respective hit view.In some embodiments, event recognizer 380 throws a flag associated withthe recognized event, and event handler 390 associated with the flagcatches the flag and performs a predefined process.

In some embodiments, event delivery instructions 388 include sub-eventdelivery instructions that deliver event information about a sub-eventwithout activating an event handler. Instead, the sub-event deliveryinstructions deliver event information to event handlers associated withthe series of sub-events or to actively involved views. Event handlersassociated with the series of sub-events or with actively involved viewsreceive the event information and perform a predetermined process.

In some embodiments, data updater 376 creates and updates data used inapplication 340-1. For example, data updater 376 stores a video fileused by media player module 344. In some embodiments, object updater 377creates and updates objects used by application 340-1. For example,object updater 377 creates a new user-interface object or updates theposition of a user-interface object. GUI updater 378 updates the GUI.For example, GUI updater 378 prepares display information and sends itto graphics module 332 for display on display system 312 (e.g., primarydisplay 102 of display portion 110, FIG. 1A and/or dynamic function row104, FIGS. 1A-1B).

In some embodiments, event handler(s) 390 includes or has access to dataupdater 376, object updater 377, and GUI updater 378. In someembodiments, data updater 376, object updater 377, and GUI updater 378are included in a single module of an application 340-1 or applicationview 391. In other embodiments, they are included in two or moresoftware modules.

It shall be understood that the foregoing discussion regarding eventhandling of user touches on touch-sensitive displays also applies toother forms of user inputs to operate electronic device 300 withinput-devices, not all of which are initiated on touch screens. Forexample, mouse movement and mouse button presses, optionally coordinatedwith single or multiple keyboard presses or holds; contact movementssuch as taps, drags, scrolls, etc., on touchpads; pen stylus inputs;movement of the device; oral instructions; detected eye movements;biometric inputs; and/or any combination thereof are optionally utilizedas inputs corresponding to sub-events which define an event to berecognized.

FIG. 4 shows a block diagram of a peripheral electronic device 400, inaccordance with some embodiments. In some embodiments, peripheralelectronic device 400 is a peripheral input and output device that atleast partially contains a dynamic function row 104 and a physical inputmechanism, such as a set of physical keys (e.g., the set of physicalkeys 106, FIGS. 2A-2B) and/or a touchpad (e.g., touchpad 108, FIGS.2B-2C), within a same housing. Examples of peripheral electronic device400 includes: peripheral keyboard (e.g., peripheral keyboard 206, FIGS.2A-2B), a peripheral touch-sensitive surface (e.g., first peripheralinput mechanism 212, FIG. 2C), or other peripheral input mechanisms(e.g., second peripheral input mechanism 222, FIG. 2D). Peripheralelectronic device 400 is communicatively coupled with computing device202 (FIGS. 2A-2D). For example, peripheral electronic device 400 iscommunicatively coupled with computing device 202 via a wiredconnection, such as USB or PS/2, or via a wireless communication link,using a communication protocol such as Bluetooth, Wi-Fi, or the like.Peripheral electronic device 400 may rely on some of the components orprocedures in electronic device 300 (FIG. 3A) or some of thesecomponents or procedures may be completed by, located in, or housed byperipheral electronic device 400 instead of electronic device 300.

In some embodiments, peripheral electronic device 400 includes one ormore of memory 402 (which optionally includes one or more computerreadable storage mediums), memory controller 422, one or more processingunits (CPU(s)) 420, peripherals interface 418, RF circuitry 408, audiocircuitry 410, speaker 411, microphone 413, input/output (I/O) subsystem406, other input or control devices 416, and external port 424.Peripheral electronic device 400 includes a touch-sensitive displaysystem 412 (e.g., dynamic function row 104, FIGS. 2A-2D) (sometimes alsoherein called a “touch-sensitive display,” a “touch screen,” or a “touchscreen display”).

Peripheral electronic device 400 optionally includes one or moreintensity sensors 465 for detecting intensity of contacts on atouch-sensitive surface such as touch-sensitive display system 412 or atouchpad (e.g., touchpad 108, FIGS. 2B-2C). Peripheral electronic device400 optionally includes one or more tactile output generators 467 forgenerating tactile outputs on a touch-sensitive surface such astouch-sensitive display system 412 or a touchpad (e.g., touchpad 108,FIGS. 2B-2C). These components optionally communicate over one or morecommunication buses or signal lines 403.

Memory 402 optionally includes high-speed random access memory andoptionally also includes non-volatile memory, such as one or moremagnetic disk storage devices, flash memory devices, or othernon-volatile solid-state memory devices. Access to memory 402 by othercomponents of peripheral electronic device 400, such as CPU(s) 420 andperipherals interface 418, is, optionally, controlled by memorycontroller 422. Peripherals interface 418 can be used to couple CPU(s)420 and memory 402 to I/O subsystem 406 and other circuitry. The one ormore processing units 420 run or execute various software programsand/or sets of instructions stored in memory 402 to perform variousfunctions for peripheral electronic device 400 and to process data. Insome embodiments, peripherals interface 418, CPU(s) 420, and memorycontroller 422 are, optionally, implemented on a single chip, such aschip 404. In some other embodiments, they are, optionally, implementedon separate chips.

RF (radio frequency) circuitry 408 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 408 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 408 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. Thewireless communication optionally uses any of a plurality ofcommunications standards, protocols and technologies, including but notlimited to near field communication (NFC), Bluetooth, Wireless Fidelity(Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and/or IEEE802.11n), Wi-MAX, or any other suitable communication protocol,including communication protocols not yet developed as of the filingdate of this document.

Optional audio circuitry 410, speaker 411, and microphone 413 provide anaudio interface between a user and peripheral electronic device 400.Audio circuitry 410 receives audio data from peripherals interface 418,converts the audio data to an electrical signal, and transmits theelectrical signal to speaker 411. Speaker 411 converts the electricalsignal to human-audible sound waves. Audio circuitry 410 also receiveselectrical signals converted by microphone 413 from sound waves. Audiocircuitry 410 converts the electrical signals to audio data andtransmits the audio data to peripherals interface 418 for processing.Audio data is, optionally, retrieved from and/or transmitted to memory402 and/or RF circuitry 408 by peripherals interface 418. In someembodiments, audio circuitry 410 also includes a headset jack. Theheadset jack provides an interface between audio circuitry 410 andremovable audio input/output peripherals, such as output-only headphonesor a headset with both output (e.g., a headphone for one or both ears)and input (e.g., a microphone).

I/O subsystem 406 couples the input/output peripherals of peripheralelectronic device 400, such as touch-sensitive display system 412 (e.g.,dynamic function row 104, FIGS. 2A-2D), to peripherals interface 418.I/O subsystem 406 optionally includes display controller 456, intensitysensor controller 459, haptic feedback controller 461, and one or moreinput controllers 460 for other input or control devices 416. The one ormore other input controllers 460 receive/send electrical signals from/toother input or control devices 416. The other input or control devices416 optionally include physical buttons (e.g., push buttons, rockerbuttons, etc.), dials, slider switches, joysticks, click wheels, a setof physical keys, a touchpad, and so forth.

Touch-sensitive display system 412 (e.g., dynamic function row 104,FIGS. 2A-2D) provides an input/output interface between peripheralelectronic device 400 and a user. Touch-sensitive display (TSD)controller 456 receives and/or sends electrical signals from/totouch-sensitive display system 412. Touch-sensitive display system 412displays visual output to the user. The visual output optionallyincludes graphics, text, icons, video, and any combination thereof(collectively termed “graphics”). In some embodiments, some or all ofthe visual output corresponds to user-interface objects/elements.

Touch-sensitive display system 412 (e.g., dynamic function row 104,FIGS. 2A-2D) includes a touch-sensitive surface, sensor, or set ofsensors that accepts input from the user based on haptic and/or tactilecontact. As such, touch-sensitive display system 412 and TSD controller456 (along with any associated modules and/or sets of instructions inmemory 402) detect contact (and any movement or breaking of the contact)on touch-sensitive display system 412 and convert the detected contactinto signals used to select or control user-interface objects (e.g., oneor more soft keys, icons, web pages, or images) that are displayed ontouch-sensitive display system 412. In one example embodiment, a pointof contact between touch-sensitive display system 412 and the usercorresponds to an area of touch-sensitive display system 412 in contactwith a finger of the user.

Touch-sensitive display system 412 (e.g., dynamic function row 104,FIGS. 2A-2D) optionally uses LCD (liquid crystal display) technology,LPD (light emitting polymer display) technology, LED (light emittingdiode) technology, or OLED (organic light emitting diode) technology,although other display technologies are used in other embodiments.Touch-sensitive display system 412 and TSD controller 456 optionallydetect contact and any movement or breaking thereof using any of aplurality of touch sensing technologies now known or later developed,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies, as well as other proximity sensorarrays or other elements for determining one or more points of contactwith touch-sensitive display system 412. In one example embodiment,projected mutual capacitance sensing technology is used, such as thatfound in the iPHONE®, iPODTOUCH®, and iPAD® from Apple Inc. ofCupertino, Calif.

Touch-sensitive display system 412 (e.g., dynamic function row 104,FIGS. 2A-2D) optionally has a video resolution in excess of 400 dpi(e.g., 500 dpi, 800 dpi, or greater). In some embodiments, the usermakes contact with touch-sensitive display system 412 using a stylus, afinger, and so forth. In some embodiments, the user interface isdesigned to work primarily with finger-based contacts and gestures.

In some embodiments, in addition to touch-sensitive display system 412,peripheral electronic device 400 optionally includes a touchpad (e.g.,touchpad 108, FIGS. 2B-2C). In some embodiments, the touchpad is atouch-sensitive area of peripheral electronic device 400 that, unliketouch-sensitive display system 412, does not display visual output. Insome embodiments, the touchpad is, optionally, a touch-sensitive surfacethat is separate from touch-sensitive display system 412, or anextension of the touch-sensitive surface formed by touch-sensitivedisplay system 412.

Peripheral electronic device 400 also includes power system 462 forpowering the various components. Power system 462 optionally includes apower management system, one or more power sources (e.g., battery,alternating current (AC), etc.), a recharging system, a power failuredetection circuit, a power converter or inverter, a power statusindicator (e.g., a light-emitting diode (LED)) and any other componentsassociated with the generation, management and distribution of power inportable devices.

Peripheral electronic device 400 optionally also includes one or morecontact intensity sensors 465 coupled with intensity sensor controller459 in I/O subsystem 406. Contact intensity sensor(s) 465 optionallyincludes one or more piezoresistive strain gauges, capacitive forcesensors, electric force sensors, piezoelectric force sensors, opticalforce sensors, capacitive touch-sensitive surfaces, or other intensitysensors (e.g., sensors used to measure the force (or pressure) of acontact on a touch-sensitive surface). Contact intensity sensor(s) 465receives contact intensity information (e.g., pressure information or aproxy for pressure information) from the environment. In someembodiments, at least one contact intensity sensor is collocated with,or proximate to, a touch-sensitive surface (e.g., touch-sensitivedisplay system 412 and/or touchpad 108, FIGS. 2B-2C).

Peripheral electronic device 400 optionally also includes one or moretactile output generators 467 coupled with haptic feedback controller461 in I/O subsystem 406. Tactile output generator(s) 467 optionallyincludes one or more electroacoustic devices such as speakers or otheraudio components and/or electromechanical devices that convert energyinto linear motion such as a motor, solenoid, electroactive polymer,piezoelectric actuator, electrostatic actuator, or other tactile outputgenerating component (e.g., a component that converts electrical signalsinto tactile outputs on the device). Contact intensity sensor(s) 465receives tactile feedback generation instructions from haptic feedbackmodule 433 and generates tactile outputs that are capable of beingsensed by a user of peripheral electronic device 400. In someembodiments, at least one tactile output generator is collocated with,or proximate to, a touch-sensitive surface (e.g., touch-sensitivedisplay system 412 and/or touchpad 108, FIGS. 2B-2C) and, optionally,generates a tactile output by moving the touch-sensitive surfacevertically (e.g., in/out of a surface of peripheral electronic device400) or laterally (e.g., back and forth in the same plane as a surfaceof peripheral electronic device 400).

In some embodiments, the software components stored in memory 402include operating system 426, communication module 428 (or set ofinstructions), contact/motion module 430 (or set of instructions), anddynamic function row module 450 (or sets of instructions). Furthermore,in some embodiments, memory 402 stores device state 457 including thedisplay state, indicating what views or other information occupy variousregions of touch-sensitive display system 412 (e.g., dynamic functionrow 104, FIGS. 2A-2D).

Operating system 426 includes various software components and/or driversfor controlling and managing general system tasks (e.g., memorymanagement, storage device control, power management, etc.) andfacilitates communication between various hardware and softwarecomponents.

Communication module 428 facilitates communication with other devices(e.g., computing device 202, FIGS. 2A-2D) over one or more externalports 424 and/or RF circuitry 408 and also includes various softwarecomponents for sending/receiving data via RF circuitry 408 and/orexternal port 424. External port 424 (e.g., Universal Serial Bus (USB),FIREWIRE, etc.) is adapted for coupling directly to other devices orindirectly over a network (e.g., the Internet, wireless LAN, etc.).

Contact/motion module 430 optionally detects contact withtouch-sensitive display system 412 and other touch sensitive devices(e.g., a touchpad or physical click wheel). Contact/motion module 430includes various software components for performing various operationsrelated to detection of contact, such as determining if contact hasoccurred (e.g., detecting a finger-down event), determining an intensityof the contact (e.g., the force or pressure of the contact or asubstitute for the force or pressure of the contact), determining ifthere is movement of the contact and tracking the movement across thetouch-sensitive surface (e.g., detecting one or more finger-draggingevents), and determining if the contact has ceased (e.g., detecting afinger-up event or a break in contact). Contact/motion module 430receives contact data from the touch-sensitive surface. Determiningmovement of the point of contact, which is represented by a series ofcontact data, optionally includes determining speed (magnitude),velocity (magnitude and direction), and/or an acceleration (a change inmagnitude and/or direction) of the point of contact. These operationsare, optionally, applied to single contacts (e.g., one finger contacts)or to multiple simultaneous contacts (e.g., “multitouch”/multiple fingercontacts). In some embodiments, contact/motion module 430 also detectscontact on a touchpad (e.g., touchpad 108, FIGS. 2B-2C).

In some embodiments, contact/motion module 430 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has selected or“clicked” on an affordance). In some embodiments at least a subset ofthe intensity thresholds are determined in accordance with softwareparameters (e.g., the intensity thresholds are not determined by theactivation thresholds of particular physical actuators and can beadjusted without changing the physical hardware of peripheral electronicdevice 400). For example, a mouse “click” threshold of a trackpad ortouch screen display can be set to any of a large range of predefinedthresholds values without changing the trackpad or touch screen displayhardware. Additionally, in some implementations a user of the device isprovided with software settings for adjusting one or more of the set ofintensity thresholds (e.g., by adjusting individual intensity thresholdsand/or by adjusting a plurality of intensity thresholds at once with asystem-level click “intensity” parameter).

Contact/motion module 430 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns (e.g., different motions, timings, and/or intensities ofdetected contacts). Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap contact includes detecting a finger-down event followed by detectinga finger-up (lift off) event at the same position (or substantially thesame position) as the finger-down event (e.g., at the position of anicon). As another example, detecting a finger swipe gesture on thetouch-sensitive surface includes detecting a finger-down event followedby detecting one or more finger-dragging events, and in some embodimentsalso followed by detecting a finger-up (lift off) event.

Haptic feedback module 433 includes various software components forgenerating instructions used by tactile output generator(s) 467 toproduce tactile outputs at one or more locations on peripheralelectronic device 400 in response to user interactions with peripheralelectronic device 400.

Dynamic function row (DFR) module 450 includes: focus obtaining module451, DFR determining module 452, and DFR presenting module 453. In someembodiments, focus obtaining module 451 is configured to obtain anindication of an active user interface element that is the current focusof the graphical user interface displayed on primary display 102 ofperipheral display device 204 (FIGS. 2A-2D) from computing device 202(FIGS. 2A-2D). In some embodiments, DFR determining module 452 isconfigured to determine graphics (e.g., a set of one or moreaffordances) based on the active user interface element that is currentfocus. Alternatively, in some embodiments, computing device 202 (FIGS.2A-2D) determines the graphics (e.g., the set of one or moreaffordances) based on the active user interface element that is in focusand provides the graphics to peripheral electronic device 400 or acomponent thereof (e.g., DFR module 450) for display on touch-sensitivedisplay system 412 (e.g., dynamic function row 104, FIGS. 2A-2D). Insome embodiments, DFR presenting module 453 is configured to render thegraphics determined by DFR determining module 452 (or provided bycomputing device 202) on touch-sensitive display system 412 (e.g.,dynamic function row 104, FIGS. 2A-2D). DFR presenting module 453includes various known software components for rendering and causingdisplay of graphics on touch-sensitive display system 412, includingcomponents for changing the visual impact (e.g., brightness,transparency, saturation, contrast or other visual property) of graphicsthat are displayed. In some embodiments, DFR module 450 includes othermodules for: adjusting the sensitivity of dynamic function row 104;adjusting the audible and/or haptic feedback provided by dynamicfunction row 104; adjusting the settings of affordances and informationdisplayed by dynamic function row 104 (e.g., size, brightness, font,language, and the like); adjusting the current power mode of dynamicfunction row 104 (e.g., normal and low-power modes); and the like.

In some embodiments, memory 402 includes event sorter 470 (e.g., inoperating system 426). In some embodiments, event sorter 470 performsthe same functions as event sorter 370 (FIG. 3B) and includes a subsetor superset of the modules, procedures, and instructions of event sorter370 (FIG. 3B). As such, event sorter 470 will not be described for thesake of brevity.

It should be appreciated that peripheral electronic device 400 is onlyan example and that peripheral electronic device 400 optionally has moreor fewer components than shown, optionally combines two or morecomponents, or optionally has a different configuration or arrangementof the components. The various components shown in FIG. 4 areimplemented in hardware, software, firmware, or a combination thereof,including one or more signal processing and/or application specificintegrated circuits.

Each of the above identified modules correspond to a set of executableinstructions for performing one or more functions described above andthe methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (i.e., sets of instructions) need notbe implemented as separate software programs, procedures or modules, andthus various subsets of these modules are, optionally, combined orotherwise re-arranged in various embodiments. In some embodiments,memory 402 optionally stores a subset of the modules and data structuresidentified above. Furthermore, memory 402 optionally stores additionalmodules and data structures not described above.

As used herein, the term “focus selector” refers to an input elementthat indicates a current part of a user interface with which a user isinteracting. In some implementations that include a cursor or otherlocation marker, the cursor acts as a “focus selector,” so that when aninput (e.g., a press input) is detected on a touch-sensitive surface(e.g., touchpad 108 in FIG. 1A of a touch-sensitive display system 412in FIG. 4) while the cursor is over a particular user interface element(e.g., a button, window, slider or other user interface element), theparticular user interface element is adjusted in accordance with thedetected input. In some implementations that include a touch-screendisplay that enables direct interaction with user interface elements onthe touch-screen display, a detected contact on the touch-screen acts asa “focus selector,” so that when an input (e.g., a press input by thecontact) is detected on the touch-screen display at a location of aparticular user interface element (e.g., a button, window, slider orother user interface element), the particular user interface element isadjusted in accordance with the detected input. In some implementations,focus is moved from one region of a user interface to another region ofthe user interface without corresponding movement of a cursor ormovement of a contact on a touch-screen display (e.g., by using a tabkey or arrow keys to move focus from one button to another button); inthese implementations, the focus selector moves in accordance withmovement of focus between different regions of the user interface.Without regard to the specific form taken by the focus selector, thefocus selector is generally the user interface element (or contact on atouch-screen display) that is controlled by the user so as tocommunicate the user's intended interaction with the user interface(e.g., by indicating, to the device, the element of the user interfacewith which the user is intending to interact). For example, the locationof a focus selector (e.g., a cursor, a contact, or a selection box) overa respective button while a press input is detected on thetouch-sensitive surface (e.g., a touchpad or touch screen) will indicatethat the user is intending to activate the respective button (as opposedto other user interface elements shown on a display of the device).

As used in the specification and claims, the term “intensity” of acontact on a touch-sensitive surface refers to the force or pressure(force per unit area) of a contact (e.g., a finger contact or a styluscontact) on the touch-sensitive surface, or to a substitute (proxy) forthe force or pressure of a contact on the touch-sensitive surface. Theintensity of a contact has a range of values that includes at least fourdistinct values and more typically includes hundreds of distinct values(e.g., at least 256). Intensity of a contact is, optionally, determined(or measured) using various approaches and various sensors orcombinations of sensors. For example, one or more force sensorsunderneath or adjacent to the touch-sensitive surface are, optionally,used to measure force at various points on the touch-sensitive surface.In some implementations, force measurements from multiple force sensorsare combined (e.g., a weighted average or a sum) to determine anestimated force of a contact. Similarly, a pressure-sensitive tip of astylus is, optionally, used to determine a pressure of the stylus on thetouch-sensitive surface. Alternatively, the size of the contact areadetected on the touch-sensitive surface and/or changes thereto, thecapacitance of the touch-sensitive surface proximate to the contactand/or changes thereto, and/or the resistance of the touch-sensitivesurface proximate to the contact and/or changes thereto are, optionally,used as a substitute for the force or pressure of the contact on thetouch-sensitive surface. In some implementations, the substitutemeasurements for contact force or pressure are used directly todetermine whether an intensity threshold has been exceeded (e.g., theintensity threshold is described in units corresponding to thesubstitute measurements). In some implementations, the substitutemeasurements for contact force or pressure are converted to an estimatedforce or pressure and the estimated force or pressure is used todetermine whether an intensity threshold has been exceeded (e.g., theintensity threshold is a pressure threshold measured in units ofpressure). Using the intensity of a contact as an attribute of a userinput allows for user access to additional device functionality that mayotherwise not be readily accessible by the user on a reduced-size devicewith limited real estate for displaying affordances (e.g., on atouch-sensitive display) and/or receiving user input (e.g., via atouch-sensitive display, a touch-sensitive surface, or aphysical/mechanical control such as a knob or a button).

In some embodiments, contact/motion module 130 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has “clicked” onan icon). In some embodiments, at least a subset of the intensitythresholds are determined in accordance with software parameters (e.g.,the intensity thresholds are not determined by the activation thresholdsof particular physical actuators and can be adjusted without changingthe physical hardware of the portable computing system 100). Forexample, a mouse “click” threshold of a trackpad or touch-screen displaycan be set to any of a large range of predefined thresholds valueswithout changing the trackpad or touch-screen display hardware.Additionally, in some implementations a user of the device is providedwith software settings for adjusting one or more of the set of intensitythresholds (e.g., by adjusting individual intensity thresholds and/or byadjusting a plurality of intensity thresholds at once with asystem-level click “intensity” parameter).

As used in the specification and claims, the term “characteristicintensity” of a contact refers to a characteristic of the contact basedon one or more intensities of the contact. In some embodiments, thecharacteristic intensity is based on multiple intensity samples. Thecharacteristic intensity is, optionally, based on a predefined number ofintensity samples, or a set of intensity samples collected during apredetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10seconds) relative to a predefined event (e.g., after detecting thecontact, prior to detecting liftoff of the contact, before or afterdetecting a start of movement of the contact, prior to detecting an endof the contact, before or after detecting an increase in intensity ofthe contact, and/or before or after detecting a decrease in intensity ofthe contact). A characteristic intensity of a contact is, optionallybased on one or more of: a maximum value of the intensities of thecontact, a mean value of the intensities of the contact, an averagevalue of the intensities of the contact, a top 10 percentile value ofthe intensities of the contact, a value at the half maximum of theintensities of the contact, a value at the 90 percent maximum of theintensities of the contact, or the like. In some embodiments, theduration of the contact is used in determining the characteristicintensity (e.g., when the characteristic intensity is an average of theintensity of the contact over time). In some embodiments, thecharacteristic intensity is compared to a set of one or more intensitythresholds to determine whether an operation has been performed by auser. For example, the set of one or more intensity thresholds mayinclude a first intensity threshold and a second intensity threshold. Inthis example, a contact with a characteristic intensity that does notexceed the first threshold results in a first operation, a contact witha characteristic intensity that exceeds the first intensity thresholdand does not exceed the second intensity threshold results in a secondoperation, and a contact with a characteristic intensity that exceedsthe second intensity threshold results in a third operation. In someembodiments, a comparison between the characteristic intensity and oneor more intensity thresholds is used to determine whether or not toperform one or more operations (e.g., whether to perform a respectiveoption or forgo performing the respective operation) rather than beingused to determine whether to perform a first operation or a secondoperation.

In some embodiments, a portion of a gesture is identified for purposesof determining a characteristic intensity. For example, atouch-sensitive surface may receive a continuous swipe contacttransitioning from a start location and reaching an end location (e.g.,a drag gesture), at which point the intensity of the contact increases.In this example, the characteristic intensity of the contact at the endlocation may be based on only a portion of the continuous swipe contact,and not the entire swipe contact (e.g., only the portion of the swipecontact at the end location). In some embodiments, a smoothing algorithmmay be applied to the intensities of the swipe contact prior todetermining the characteristic intensity of the contact. For example,the smoothing algorithm optionally includes one or more of: anunweighted sliding-average smoothing algorithm, a triangular smoothingalgorithm, a median filter smoothing algorithm, and/or an exponentialsmoothing algorithm. In some circumstances, these smoothing algorithmseliminate narrow spikes or dips in the intensities of the swipe contactfor purposes of determining a characteristic intensity.

In some embodiments one or more predefined intensity thresholds are usedto determine whether a particular input satisfies an intensity-basedcriterion. For example, the one or more predefined intensity thresholdsinclude (i) a contact detection intensity threshold IT₀, (ii) a lightpress intensity threshold IT_(L), (iii) a deep press intensity thresholdIT_(D) (e.g., that is at least initially higher than I_(L)), and/or (iv)one or more other intensity thresholds (e.g., an intensity thresholdI_(H) that is lower than I_(L)). In some embodiments, the light pressintensity threshold corresponds to an intensity at which the device willperform operations typically associated with clicking a button of aphysical mouse or a trackpad. In some embodiments, the deep pressintensity threshold corresponds to an intensity at which the device willperform operations that are different from operations typicallyassociated with clicking a button of a physical mouse or a trackpad. Insome embodiments, when a contact is detected with a characteristicintensity below the light press intensity threshold (e.g., and above anominal contact-detection intensity threshold IT₀ below which thecontact is no longer detected), the device will move a focus selector inaccordance with movement of the contact on the touch-sensitive surfacewithout performing an operation associated with the light pressintensity threshold or the deep press intensity threshold. Generally,unless otherwise stated, these intensity thresholds are consistentbetween different sets of user interface figures.

In some embodiments, the response of the device to inputs detected bythe device depends on criteria based on the contact intensity during theinput. For example, for some “light press” inputs, the intensity of acontact exceeding a first intensity threshold during the input triggersa first response. In some embodiments, the response of the device toinputs detected by the device depends on criteria that include both thecontact intensity during the input and time-based criteria. For example,for some “deep press” inputs, the intensity of a contact exceeding asecond intensity threshold during the input, greater than the firstintensity threshold for a light press, triggers a second response onlyif a delay time has elapsed between meeting the first intensitythreshold and meeting the second intensity threshold. This delay time istypically less than 200 ms in duration (e.g., 40, 100, or 120 ms,depending on the magnitude of the second intensity threshold, with thedelay time increasing as the second intensity threshold increases). Thisdelay time helps to avoid accidental deep press inputs. As anotherexample, for some “deep press” inputs, there is a reduced-sensitivitytime period that occurs after the time at which the first intensitythreshold is met. During the reduced-sensitivity time period, the secondintensity threshold is increased. This temporary increase in the secondintensity threshold also helps to avoid accidental deep press inputs.For other deep press inputs, the response to detection of a deep pressinput does not depend on time-based criteria.

In some embodiments, one or more of the input intensity thresholdsand/or the corresponding outputs vary based on one or more factors, suchas user settings, contact motion, input timing, application running,rate at which the intensity is applied, number of concurrent inputs,user history, environmental factors (e.g., ambient noise), focusselector position, and the like. Example factors are described in U.S.patent application Ser. Nos. 14/399,606 and 14/624,296, which areincorporated by reference herein in their entireties.

For example, FIG. 3C illustrates a dynamic intensity threshold 480 thatchanges over time based in part on the intensity of touch input 476 overtime. Dynamic intensity threshold 480 is a sum of two components, firstcomponent 474 that decays over time after a predefined delay time p1from when touch input 476 is initially detected, and second component478 that trails the intensity of touch input 476 over time. The initialhigh intensity threshold of first component 474 reduces accidentaltriggering of a “deep press” response, while still allowing an immediate“deep press” response if touch input 476 provides sufficient intensity.Second component 478 reduces unintentional triggering of a “deep press”response by gradual intensity fluctuations of in a touch input. In someembodiments, when touch input 476 satisfies dynamic intensity threshold480 (e.g., at point 481 in FIG. 3C), the “deep press” response istriggered.

FIG. 3D illustrates another dynamic intensity threshold 486 (e.g.,intensity threshold I_(D)). FIG. 3D also illustrates two other intensitythresholds: a first intensity threshold I_(H) and a second intensitythreshold I_(L). In FIG. 3D, although touch input 484 satisfies thefirst intensity threshold I_(H) and the second intensity threshold I_(L)prior to time p2, no response is provided until delay time p2 haselapsed at time 482. Also in FIG. 3D, dynamic intensity threshold 486decays over time, with the decay starting at time 488 after a predefineddelay time p1 has elapsed from time 482 (when the response associatedwith the second intensity threshold I_(L) was triggered). This type ofdynamic intensity threshold reduces accidental triggering of a responseassociated with the dynamic intensity threshold I_(D) immediately after,or concurrently with, triggering a response associated with a lowerintensity threshold, such as the first intensity threshold I_(H) or thesecond intensity threshold I_(L).

FIG. 3E illustrate yet another dynamic intensity threshold 492 (e.g.,intensity threshold I_(D)). In FIG. 3E, a response associated with theintensity threshold I_(L) is triggered after the delay time p2 haselapsed from when touch input 490 is initially detected. Concurrently,dynamic intensity threshold 492 decays after the predefined delay timep1 has elapsed from when touch input 490 is initially detected. So adecrease in intensity of touch input 490 after triggering the responseassociated with the intensity threshold I_(L), followed by an increasein the intensity of touch input 490, without releasing touch input 490,can trigger a response associated with the intensity threshold I_(D)(e.g., at time 494) even when the intensity of touch input 490 is belowanother intensity threshold, for example, the intensity threshold I_(L).

An increase of characteristic intensity of the contact from an intensitybelow the light press intensity threshold IT_(L) to an intensity betweenthe light press intensity threshold IT_(L) and the deep press intensitythreshold ITS is sometimes referred to as a “light press” input. Anincrease of characteristic intensity of the contact from an intensitybelow the deep press intensity threshold ITS to an intensity above thedeep press intensity threshold ITS is sometimes referred to as a “deeppress” input. An increase of characteristic intensity of the contactfrom an intensity below the contact-detection intensity threshold IT₀ toan intensity between the contact-detection intensity threshold IT₀ andthe light press intensity threshold IT_(L) is sometimes referred to asdetecting the contact on the touch-surface. A decrease of characteristicintensity of the contact from an intensity above the contact-detectionintensity threshold IT₀ to an intensity below the contact-detectionintensity threshold IT₀ is sometimes referred to as detecting liftoff ofthe contact from the touch-surface. In some embodiments IT₀ is zero. Insome embodiments, IT₀ is greater than zero. In some illustrations ashaded circle or oval is used to represent intensity of a contact on thetouch-sensitive surface. In some illustrations, a circle or oval withoutshading is used represent a respective contact on the touch-sensitivesurface without specifying the intensity of the respective contact.

In some embodiments, described herein, one or more operations areperformed in response to detecting a gesture that includes a respectivepress input or in response to detecting the respective press inputperformed with a respective contact (or a plurality of contacts), wherethe respective press input is detected based at least in part ondetecting an increase in intensity of the contact (or plurality ofcontacts) above a press-input intensity threshold. In some embodiments,the respective operation is performed in response to detecting theincrease in intensity of the respective contact above the press-inputintensity threshold (e.g., the respective operation is performed on a“down stroke” of the respective press input). In some embodiments, thepress input includes an increase in intensity of the respective contactabove the press-input intensity threshold and a subsequent decrease inintensity of the contact below the press-input intensity threshold, andthe respective operation is performed in response to detecting thesubsequent decrease in intensity of the respective contact below thepress-input threshold (e.g., the respective operation is performed on an“up stroke” of the respective press input).

In some embodiments, the device employs intensity hysteresis to avoidaccidental inputs sometimes termed “jitter,” where the device defines orselects a hysteresis intensity threshold with a predefined relationshipto the press-input intensity threshold (e.g., the hysteresis intensitythreshold is X intensity units lower than the press-input intensitythreshold or the hysteresis intensity threshold is 75%, 90%, or somereasonable proportion of the press-input intensity threshold). Thus, insome embodiments, the press input includes an increase in intensity ofthe respective contact above the press-input intensity threshold and asubsequent decrease in intensity of the contact below the hysteresisintensity threshold that corresponds to the press-input intensitythreshold, and the respective operation is performed in response todetecting the subsequent decrease in intensity of the respective contactbelow the hysteresis intensity threshold (e.g., the respective operationis performed on an “up stroke” of the respective press input).Similarly, in some embodiments, the press input is detected only whenthe device detects an increase in intensity of the contact from anintensity at or below the hysteresis intensity threshold to an intensityat or above the press-input intensity threshold and, optionally, asubsequent decrease in intensity of the contact to an intensity at orbelow the hysteresis intensity, and the respective operation isperformed in response to detecting the press input (e.g., the increasein intensity of the contact or the decrease in intensity of the contact,depending on the circumstances).

For ease of explanation, the description of operations performed inresponse to a press input associated with a press-input intensitythreshold or in response to a gesture including the press input are,optionally, triggered in response to detecting: an increase in intensityof a contact above the press-input intensity threshold, an increase inintensity of a contact from an intensity below the hysteresis intensitythreshold to an intensity above the press-input intensity threshold, adecrease in intensity of the contact below the press-input intensitythreshold, or a decrease in intensity of the contact below thehysteresis intensity threshold corresponding to the press-inputintensity threshold. Additionally, in examples where an operation isdescribed as being performed in response to detecting a decrease inintensity of a contact below the press-input intensity threshold, theoperation is, optionally, performed in response to detecting a decreasein intensity of the contact below a hysteresis intensity thresholdcorresponding to, and lower than, the press-input intensity threshold.As described above, in some embodiments, the triggering of theseresponses also depends on time-based criteria being met (e.g., a delaytime has elapsed between a first intensity threshold being met and asecond intensity threshold being met).

User Interfaces and Associated Processes

Attention is now directed towards embodiments of user interfaces (“UIs”)and associated processes that may be implemented by portable computingsystem 100 (FIG. 1A) or desktop computing system 200 (FIGS. 2A-2D). Insome embodiments, primary display 102 is implemented in display portion110 of portable computing system 100 (FIG. 1A). Alternatively, in someembodiments, primary display 102 is implemented in peripheral displaydevice 204 (FIGS. 2A-2D). In some embodiments, dynamic function row 104is a touch-sensitive secondary display implemented in body portion 120of portable computing system 100 (FIGS. 1A-1B). Alternatively, in someembodiments, dynamic function row 104 is a touch-sensitive secondarydisplay implemented in peripheral keyboard 206 (FIGS. 2A-2B), firstperipheral input mechanism 212 (FIG. 2C), or second peripheral inputmechanism 222 (FIG. 2D).

FIGS. 5A-5N are schematics of primary and secondary displays used toillustrate example user interfaces for enabling low-vision users tointeract with touch-sensitive secondary displays, in accordance withsome embodiments. The user interfaces in these figures are used toillustrate the methods and/or processes described below, including themethods in FIGS. 6A-6C, 7A-7C, and 8A-8C. One of ordinary skill in theart will appreciate that the following user interfaces are merelyexamples. Moreover, one of ordinary skill in the art will appreciatethat additional affordances and/or user interface elements, or thatfewer affordances and/or user interface elements may be used inpractice.

FIG. 5A illustrates primary display 102 displaying a user interface fora system preferences application, in which an Accessibility option for“Enable TouchBar Zoom” is presented. As is also shown in FIG. 5A,affordances displayed at the touch-sensitive secondary display 104 areused to activate or control application-specific functionalityassociated with the system preferences application that is currentlydisplayed at the primary display 102. For example, the touch-sensitivesecondary display includes application-specific affordances for enablingtouchbar zoom, navigating between menu options (affordances next to thesystem-level “esc” affordance), and a “show all” affordance for causingthe primary display to exit the Accessibility menu and instead presentall of the system preference menu options. FIG. 5A also illustratesthat, in addition to these application-specific affordances, thetouch-sensitive secondary display 104 includes system-level affordances5085-5088 for controlling system-level functions (as discussed below inreference to method 600).

FIG. 5A shows user selection of the enable touchbar zoom checkbox usinga cursor 504 (in some embodiments, the primary display is alsotouch-sensitive and, thus, a user may provide an input using theirfinger or a stylus instead of using the cursor 504). In response to userselection of the checkbox, the touchbar zoom accessibility mode is thenenabled for the touch-sensitive secondary display 104 (as shown be theselected checkbox at both the primary and the touch-sensitive secondarydisplays in FIG. 5B). In some embodiments, an accessibility mode is amode of operation in which certain features are activated to enableusers with vision, hearing, physical and motor skills, or learning andliteracy impairments to interact with their electronic devices (i.e., toallow people with disabilities to drive a user interface innon-traditional ways), and these certain features are not available (ormay be disabled) during a normal mode of operation for electronicdevices. For example, as described herein, a touchbar zoom accessibilitymode is a mode of operation for a touch-sensitive secondary display inwhich interactions at the touch-sensitive secondary display cause azoomed-in representation of affordances and user interfaces displayed atthe touch-sensitive secondary display to be presented at a primarydisplay, thereby enabling low-vision users of the touch-sensitivesecondary display to view and interact with these affordances and userinterfaces. Stated another way, activating the touchbar zoomaccessibility mode allows people with vision impairments to drive userinterfaces presented at the touch-sensitive secondary display in anon-traditional way, i.e., by viewing a zoomed-in representation on aprimary display and providing selection and other inputs at thetouch-sensitive secondary display. In some embodiments, the touchbarzoom accessibility mode is disabled by default and must be activatedusing, e.g., the user interfaces shown in FIGS. 5A-5B.

FIG. 5B also illustrates user selection of a mail icon 506, using cursor504, from within an app tray 514 that includes affordances 506-515 foractivating various applications. In response to the user selection ofthe mail icon 506, a user interface for the mail/email application 580is then presented on the primary display 102 and the touch-sensitivesecondary display 104 is updated to include new application-specificaffordances that correspond to functions available within the mailapplication 580 (e.g., affordances 5080-5084).

FIG. 5C also illustrates a user selecting text “Everyone,” using cursor504, within the body of an email that is displayed at the user interfacefor the email application 580. As shown in FIG. 5D, in response to theuser selecting this text, the text is then highlighted at the primarydisplay 102 and the touch-sensitive secondary display is updated toinclude application-specific affordances that are used to manipulate howthe selected text is rendered or displayed (e.g., affordances 5102,5106, and 5089-5091).

FIG. 5D also illustrates an input 5102 at the touch-sensitive secondarydisplay 104 that contacts the affordance 5104 and, in response to theinput 5102, the primary display is updated to present a zoomed-inrepresentation of the contacted affordance 5104 (e.g., zoomed-inrepresentation 5204) that is shown within the user interface 5202 thatincludes zoomed-in representations of other affordances displayed at thesecondary display 104. Additionally, a focus indicator 5206 is presentedat the user interface 5202 that is positioned to correspond to theposition of the input 5102 at the secondary display 104. By providinglarger affordances and a focus indicator, in response to just a singleinput at the secondary display 104, low-vision users are enabled tointeract with (and have sustained interactions with) the touch-sensitivesecondary display 104 (as explained in more detail below in reference tothe methods 600-800.

FIG. 5D also shows that as the input 5102 begins moving laterally acrossthe secondary display 104 and, in response, the user interface 5202 isshifted in accordance with this lateral movement and to reveal otherzoomed-in representations of affordances on the primary display (asshown in FIG. 5E, the user interface 5202 shifts to the right to revealadditional zoomed-in affordances). FIG. 5E shows a second input 5103that contacts the touch-sensitive secondary display 104 (in response asecond focus indicator is presented at the zoomed-in user interface 5202that is positioned based on a position of the second input 5103 at thesecondary display 104). As shown in FIG. 5E, the input 5102 and thesecond input 5103 then are used to provide a de-pinch gesture in whichthe two inputs move in substantially opposite directions across thesecondary display 104 and, in response to this de-pinch gesture, thezoom level of the zoomed-in user interface 5202 is increased (as shownin FIG. 5F). In some embodiments, a user also presses a key, such as a“Command” key before providing the de-pinch gesture, to modify the zoomlevel of the zoomed-in user interface via a de-pinch gesture.

In some embodiments, users are also able to decrease the zoom level byproviding a pinch gesture (as shown for inputs 5102 and 5103 in FIG. 5F)and, in response, a zoom level of the zoomed-in user interface 5202 isdecreased accordingly (as shown in FIG. 5G). FIG. 5G then illustratesinput 5102 moving laterally across the secondary display 104 and back toits original position (FIG. 5H) that was shown in FIG. 5D. FIG. 5Hillustrates that a user may activate the affordance 5104 by providing asplit-tap gesture, e.g., maintaining input 5102 at the secondary display104 and also providing momentary tap 5106 at the secondary display 104.In response, an appearance of the focus indicator 5206 may change (FIG.5K) and the affordances 5104 and 5204 are then movable across theirrespective associated sliders.

Alternatively, instead of providing the split-tap gesture, a user maymaintain contact with affordance 5104 for more than a predeterminedperiod of time (e.g., 0.5 seconds, 0.75 seconds, or 1 second) and, inresponse, a representation of a countdown timer 5208 is updated to beingcounting down until the affordance 5104 is activated (as shown in FIGS.5I-5K in which the countdown timer that surrounds the focus indicator5206 moves in a clockwise direction until it expires and the affordances5104 and 5204 are then movable).

As shown in FIG. 5L, a user may then move the affordances 5104 and 5204across their respective sliders to manipulate a text size associatedwith the selected “Everyone” text. For example, as the user moves theaffordances 5104/5204 across the sliders, the text size for “Everyone”is dynamically increased in accordance with the movement across thesliders (FIG. 5L).

Turning now to FIG. 5M, an additional example of a split-tap gesture isprovided, in which the input 5102 remains in contact with an affordance(e.g., affordance 5089 for bolding text in the email application) and amomentary tap 5107 is detected that is not over the affordance. Inresponse to this split-tap gesture, the affordance contacted by theinput 5102 (e.g., affordance 5089) is activated and a functionassociated with that affordance is performed (e.g., as shown in FIG. 5N,the selected text “Everyone” now appears bolded). FIG. 5N alsoillustrates that input 5102 has lifted-off from the touch-sensitivesecondary display 104 and, in response, the zoomed-in representationsare no longer presented at the primary display 102.

Additional descriptions regarding FIGS. 5A-5N are provided below inreferences to methods 600-800.

FIGS. 6A-6C are a flowchart of a method 600 that enables low-visionusers to interact with touch-sensitive secondary displays, in accordancewith some embodiments. The method 600 is performed (602) at a computingsystem including one or more processors, memory, a first housingincluding a primary display, and a second housing at least partiallycontaining a touch-sensitive secondary display that is distinct from theprimary display. Some operations in method 600 are, optionally, combinedand/or the order of some operations is, optionally, changed.

In some embodiments, the computing system is portable computing system100 (FIG. 1A) or desktop computing system 200 (FIGS. 2A-2D). In someembodiments, the primary display is primary display 102 (FIG. 1A) whichis implemented in display portion 110 (also referred to herein as afirst housing 110 that includes the primary display 102) of portablecomputing system 100 (FIG. 1A) and the second housing is a body portion120 of the portable computing system 100, and the second housing atleast partially contains the touch-sensitive secondary display (e.g.,dynamic function row 104, FIGS. 1A-1B) and a physical keyboard (e.g.,the set of physical keys 106) (604).

In some embodiments, the second housing is not connected to the firsthousing (606), e.g., because the first housing is part of a first deviceand the second housing is part of a different device other than thefirst device (e.g., the second housing could be part of a mobile phone,a tablet device, or any other device that includes affordances that aredisplayed on a smaller secondary display while that other device isconnected to a computing system that includes a larger primary display).As one non-limiting example, in some embodiments, the second housing isperipheral keyboard 206 (FIGS. 2A-2B) of desktop computing system 200,which at least partially contains the touch-sensitive secondary display(e.g., dynamic function row 104, FIGS. 2A-2B) and the physical keyboard(e.g., the set of physical keys 106, FIGS. 2A-2B). As another example,in some embodiments, the second housing is first peripheral inputmechanism 212 (FIG. 2C) of desktop computing system 200, which at leastpartially contains the touch-sensitive secondary display (e.g., dynamicfunction row 104, FIG. 2C) and the second housing includes an inputmechanism (e.g., touchpad 108, FIG. 2C) and does not include thephysical keyboard. As one more example, in some embodiments, the secondhousing is part of a wearable computing device (608), such as a smartwatch.

As described below, the method 600 (and associated interfaces) enableslow-vision users to interact with touch-sensitive secondary displays. Asshown in FIG. 6A, the method 600 includes displaying (610), on theprimary display, a first user interface for an application (e.g., asshown in FIG. 5C, a first user interface for an email application isshown as displayed on the primary display 102). The method 600 alsoincludes: displaying (612), on the touch-sensitive secondary display, asecond user interface that includes a plurality of application-specificaffordances that control functions available within the application, andeach of the plurality of application-specific affordances is displayedwith a first display size. For example, the plurality ofapplication-specific affordances are selected to controlcontextually-relevant functions in the application, such as functionsthat are a user would need to access based on what they are currentlydoing within the application (such as the plurality ofapplication-specific affordances shown in FIG. 5C that may be used tosend an email, add emoticons to an email message, and to select variousautocomplete options). In some embodiments or circumstances, theplurality of application-specific affordances also changes based on theuser's interactions with the application (e.g., FIG. 5C shows that theuser is selecting text in an email message using cursor 504 and, inresponse to selection of the text, the plurality of application-specificaffordances that are displayed in the touch-sensitive secondary display104 is updated to include text-editing options, such as slider knob 5104for editing text size by dragging it along slider 5106 and affordances5082-5084 for selecting bold, italic, and underline options, as shown inFIG. 5D). Additional details and numerous examples regarding howaffordances may change at a touch-sensitive secondary display based on auser's interactions at a primary display are provided in commonly-ownedU.S. patent application Ser. No. 15/275,298, which is herebyincorporated by reference in its entirety.

In some embodiments, and as shown in FIG. 5C, the touch-sensitivesecondary also includes one or more system-level affordances foractivating or controlling system-level functions (e.g., affordance 5079for performing an escape function, affordance 5085 for controllingbrightness of the primary display 102, affordance 5086 for controllingvolume levels for the computing system, affordance 5087 for muting soundat the computing system, and affordance 5088 for controlling oractivating a virtual personal assistant available via the computingsystem).

In some embodiments, each of the application-specific affordances is(only) selectable via one or more inputs at the touch-sensitivesecondary display (614). Stated another way, the affordances (any of theplurality of application-specific affordances and any system-levelaffordances) may not be selected in any other way except by providinginputs at the secondary display. In some embodiments, the one or moreinputs include a quick tap that includes a liftoff of the input followedby a tap shortly thereafter, a split-tap (as described below), and aselection event that occurs after expiration of a countdown timer (asdescribed below).

The method 600 also includes detecting (616), via the touch-sensitivesecondary display, an input that contacts at least oneapplication-specific affordance of the plurality of application-specificaffordances (e.g., input 5102 at affordance 5104, FIG. 5D). In responseto detecting the input and while the input remains in contact with thetouch-sensitive secondary display, the method 600 includes (618)continuing to display, on the primary display, the first user interfacefor the application (e.g., the primary display continues to display thefirst user interface for the email application) and displaying, on theprimary display, a zoomed-in representation of the at least oneapplication-specific affordance (e.g., zoomed-in representation 5204).The zoomed-in representation of the at least one application-specificaffordance is displayed with a second display size that is larger thanthe first display size (as shown in FIG. 5D affordance 5204 is shownwith a larger display size than affordance 5104).

In some instances, users of computing systems (in particular, low-visionusers) are unable to accurately view icons or affordances that aredisplayed with a small display size (such as those shown on a smartwatch). Populating a touch-sensitive secondary display withapplication-specific affordances and then displaying a zoomed-inrepresentation of one of those affordances at a larger, primary displayin response to a single input (as explained above) provides these userswith clear visual feedback indicating which affordance they may beselecting. Providing this improved visual feedback to the user enhancesoperability of the device and makes the human-machine interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with affordancesdisplayed on the secondary display). Additionally, allowing these usersto accurately view affordances displayed on a small screen enables asustained interaction with the touch-sensitive secondary display thatwould not otherwise be possible due to frequent mistakes (e.g.,incorrect selections) and the need to waste time correcting thesemistakes.

In some embodiments, displaying the zoomed-in representation of the atleast one application-specific affordance includes (620) displaying thezoomed-in representation of the at least one application-specificaffordance within a zoomed-in representation of the second userinterface (e.g., zoomed-in affordance 5204 is displayed within zoomed-inrepresentation 5202 of the second user interface). In some embodiments,displaying the zoomed-in representation of the second user interfaceincludes displaying on the primary display a zoomed-in representation ofa second application-specific affordance that is adjacent to theapplication-specific affordance on the touch-sensitive secondarydisplay.

Displaying a zoomed-in view of the at least one application-specificaffordance within a zoomed-in representation of the second userinterface in response to a single input provides users with clear visualfeedback indicating which affordance they may be selecting andindicating which affordances are located in proximity thereto. Providingthis improved visual feedback to the user enhances operability of thedevice and makes the human-machine interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with affordances and other neighboringaffordances displayed on the secondary display). Additionally, allowingthese users to accurately view affordances displayed on a small screenenables a sustained interaction with the touch-sensitive secondarydisplay that would not otherwise be possible due to frequent mistakes(e.g., incorrect selections) and the need to waste time correcting thesemistakes.

In some other embodiments, instead of (or in addition to) displaying thezoomed-in representation of the at least one application-specificaffordance within the zoomed-in representation of the second userinterface, the zoomed-in representation of the at least oneapplication-specific affordance is displayed in a central region of theprimary display (e.g., overlaying the first user interface for the emailapplication).

Turning now to FIG. 6B, in some embodiments, displaying the zoomed-inrepresentation of the second user interface includes (622) displaying afocus indicator (e.g., focus indicator 5206, including a representationof a countdown timer 5208, FIG. 5D) within the zoomed-in representationof the second user interface that is positioned based at least in parton a position on the touch-sensitive secondary display at which theinput contacted the touch-sensitive secondary display (e.g., theposition of focus indicator 5206 within zoomed-in representation 5202generally corresponds to the position of input 5102 at thetouch-sensitive secondary display 104). In some embodiments, the focusindicator is displayed in response to the input and provides low-visionusers with a clear visual indication as to where their fingers arelocated on the touch-sensitive secondary display, to allow these usersto accurately select displayed affordances from within thetouch-sensitive secondary display. Displaying a focus indicator withinthe zoomed-in representation of the second user interface provides userswith clear visual feedback as to the location of their finger on thetouch-sensitive secondary display. In some instances, the users may notbe able to see small affordances displayed on the touch-sensitivesecondary display due to vision problems or due to their fingerobscuring affordances located underneath. Therefore, providing the focusindicator within the zoomed-in representation enhances operability ofthe device and makes the human-machine interface more efficient (e.g.,by helping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with affordances and other neighboringaffordances displayed on the secondary display). Additionally, allowingthese users to accurately understand the location of their finger on thetouch-sensitive secondary display enables a sustained interaction withthe touch-sensitive secondary display that would not otherwise bepossible due to frequent mistakes (e.g., incorrect selections) and theneed to waste time correcting these mistakes.

In some embodiments, the focus indicator includes (624) a representationof a countdown timer (e.g., representation 5208 of a countdown timerthat encircles a focus indicator 5206). In some embodiments, the method600 includes: in accordance with a determination that the input hasremained in contact with the at least one application-specificaffordance for more than the predetermined amount of time (thisdetermination may also include determining that the user has not movedthe input beyond a threshold distance (e.g., 5 px) over a thresholdperiod of time (e.g., 1 second)), updating the representation of thecountdown timer to indicate that the countdown timer is active. Forexample, as shown in FIGS. 5H-5K, the representation 5208 of thecountdown timer that encircles the focus indicator 5206 moves in aclockwise direction to reflect that the timer is counting down. Inaccordance with a determination that the countdown timer has expired(e.g., after a period of 1, 1.5, or 2 seconds), the method 600 includes:activating the at least one application-specific affordance. Displayinga focus indicator with a representation of a countdown timer that beginscounting down after an input has remained in contact with an affordancefor more than a predetermined period of time provides users with clearvisual feedback that they are about to select an affordance. In someinstances, the users may not be able to see small affordances displayedon the touch-sensitive secondary display due to vision problems or dueto their finger obscuring affordances located underneath and, thus,these users may not realize when they are activating/selecting variousaffordances. Therefore, providing the focus indicator with the countdowntimer enhances operability of the device and makes the human-machineinterface more efficient (e.g., by helping the user to provide properinputs and reducing user mistakes when operating/interacting withaffordances and other neighboring affordances displayed on the secondarydisplay). Additionally, allowing these users to accurately understandwhen they are about to activate an affordance enables a sustainedinteraction with the touch-sensitive secondary display that would nototherwise be possible due to frequent mistakes (e.g., incorrectselections) and the need to waste time correcting these mistakes.

In some embodiments, activating the application-specific affordance mayinclude any of the following: (i) performing a function that isassociated with the at least one application-specific affordance (suchas a function available via the application); (ii) expanding theaffordance within the touch-sensitive secondary display and the primarydisplay to reveal additional functionality (such as expanding a volumeaffordance to display a slider that allows for modifying volumesettings); and (iii) allowing a user to move the affordance along aslider. In some embodiments, and as explained below, activating the atleast one application-specific affordance may be performed based onreceiving other types of inputs. For example, activation of theaffordance may occur in accordance with a determination that a quick tapgesture has been received over the at least one application-specificaffordance or that a split-tap has been received that selects the atleast one application-specific affordance (i.e., a first input is overthe affordance and a second tap is not over the affordance but causesselection of the affordance).

In some embodiments, the at least one application-specific affordance isassociated (626) with a slider (e.g., affordance 5104 is associated withslider 5106), and activating the at least one application-specificaffordance includes updating the zoomed-in representation of the atleast one application-specific affordance in accordance with (or, moregenerally, to allow) movement of the at least one application-specificaffordance along the slider (e.g., as shown in FIG. 5K affordance 5204is now movable along the slider, in response to the countdown timer offocus indicator 5206 having expired). In some embodiments, the at leastone application-specific affordance is not slid-able prior to it beingactivated after expiration of the countdown timer (or after itsactivation in response to the quick tap and split-tap inputs discussedabove). In some embodiments, the touch-sensitive secondary display isalso updated to allow this same movement of the at least oneapplication-specific affordance (e.g., affordance 5104 now moves alongslider 5106). In some embodiments, after the expiration of the countdowntimer, a visual characteristic used to render (or that is associatedwith) the focus indicator is modified to indicate that the focusindicator is now able to move the at least one application-specificaffordance (e.g., as shown in FIGS. 5K-5L, the representation of thecountdown timer remains in its expired state to indicate that theaffordance located underneath is now movable/slid-able).

Activating an affordance that is associated with a slider afterexpiration of a countdown timer helps to enhance operability of thedevice and makes the human-machine interface more efficient (e.g., byreducing user mistakes when operating/interacting with affordances andhelping to avoid accidental modification of a slider). Additionally,allowing users to accurately understand when they are able to manipulatea slider enables a sustained interaction with the touch-sensitivesecondary display that would not otherwise be possible due to frequentmistakes (e.g., incorrect or accidental manipulations of a slider) andthe need to waste time correcting these mistakes.

In some embodiments, the zoomed-in representation of the at least oneapplication-specific affordance is displayed (628) on the primarydisplay in accordance with a determination that the input has remainedin continuous contact with the touch-sensitive secondary display formore than a predetermined amount of time (e.g., 0.5 seconds, 0.75seconds, or 1 second pass, while the input 5012 remains in continuouscontact with the touch-sensitive secondary display 104, before thezoomed-in representation is presented at the primary display). In someembodiments, if the input remains in such continuous contact with thetouch-sensitive secondary display then the device determines that theinput is not a selection or tap input and, based on that determination,the devices then presents the zoomed-in representation of the at leastone application-specific affordance at the primary display (in someembodiments, and as discussed below, the device conducts thisdetermination only when the touch-sensitive secondary display isoperating in an accessibility mode).

In some embodiments, the zoomed-in representation of the at least oneapplication-specific affordance is displayed (630) in accordance with adetermination that the touch-sensitive secondary display is operating inan accessibility mode. (e.g., the accessibility mode is activated byselecting a system preference, as shown in FIG. 5A-5B, and as explainedin more detail below in reference to method 700).

With reference now to FIG. 6C, the method 600, in some embodiments,includes: while the input remains in contact with the at least oneapplication-specific affordance at the touch-sensitive secondarydisplay, detecting (632) a tap gesture (e.g., tap 5105, FIG. 5H) at thetouch-sensitive secondary display that does not contact the at least oneapplication-specific affordance. In some embodiments, thetouch-sensitive secondary display includes a first area comprising theat least one application-specific affordance and a second areacomprising other affordances in the plurality of application-specificaffordances (i.e., these other affordances do not include the at leastone application-specific affordance), and the tap gesture is received atthe second area and thus does not contact the at least oneapplication-specific affordance. In some embodiments, this “split-tapgesture” is an alternative selection option (instead of having to waitfor a countdown timer to expire). In response to detecting the tapgesture, the method 600 includes: activating the at least oneapplication-specific affordance (e.g., allowing a user to moveaffordances 5204/5104 along respective sliders, as explained above inreference to FIG. 5K-5L). Another example is shown in FIGS. 5M-5N, inwhich the input 5102 remains in contact with the touch-sensitivesecondary display at affordance 5090 and then tap gesture 5107 isreceived that is not over the affordance 5090 and, in response, theaffordance 5090 is activated which causes the selected text “Everyone”at the primary display 102 to be bolded. Allowing activation of anaffordance that is in contact with an input in response to a tap gesturethat is not over the affordance (the split-tap gesture) enhancesoperability of the device and makes the human-machine interface moreefficient (e.g., by allowing users to place a first finger over adesired affordance and then use a different finger to perform aselection of that desired affordance). Additionally, allowing users tomove their first finger freely around the touch-sensitive secondarydisplay allows users to maintain a sustained interaction with thetouch-sensitive secondary display (by exploring which affordances aredisplayed at the touch-sensitive secondary display), that would nototherwise be possible due to frequent mistakes (e.g., incorrect oraccidental selections of affordances) and the need to waste timecorrecting these mistakes.

In some embodiments, users are also able to activate various types ofaffordances using the split-tap gesture. For example, in someembodiments, users are able to manipulate date or time ranges forcalendar events by first placing their fingers at either end of thedate-range-modification affordance (such as the examples shown in FIGS.36J-36Q of commonly-owned patent application Ser. No. 15/275,298,incorporated by reference above). In some embodiments, the primarydisplay is then updated to include focus indicators with countdowntimers at each end of the date-range-modification affordance and theuser is able to manipulate the date range upon expiration of both of thecountdown timers (as shown in FIGS. 5E-5F, focus indicators 5206 and5212 are provided with each input at the touch-sensitive secondarydisplay). In some embodiments, users may modify one end of an affordancethat includes a range of values by allowing the countdown timer of afocus indicator to expire at either end of the affordance (e.g., tomanipulate just one end of the affordance).

In some embodiments, the method 600 includes: detecting (634), at thetouch-sensitive secondary display, a predefined gesture (e.g., userpresses command button and then performs a pinch gesture with inputs5102 and 5103 (FIG. 5F) or de-pinch gesture with inputs 5102 and 5103(FIG. 5E) at the touch-sensitive secondary display) that manipulates azoom level that is used to display the zoomed-in representation of theat least one application-specific affordance at the primary display. Inresponse to detecting the predefined gesture (or in response todetecting each incremental manipulation of the zoom level during thepredefined gesture), the method 600 includes: updating the zoomed-inrepresentation at the primary display as the zoom level is manipulatedusing the predefined gesture (e.g., zooming-in further in response tothe de-pinch gesture of FIG. 5E or zooming-out in response to the pinchgesture of FIG. 5F). Allowing users to manipulate a zoom level for thezoomed-in representation using a predefined gesture enhances operabilityof the device and makes the human-machine interface more efficient(e.g., by allowing users to quickly and easily adjust the zoom level tosuit their personal preferences). Additionally, allowing users tomanipulate the zoom level allows users to maintain a sustainedinteraction with the touch-sensitive secondary display by ensuring thatthese users are able to adjust the zoom level so that they are able toaccurately view affordances that may be displayed at different displaysizes, which is important for low-vision users of various sight levels.

It should be understood that the particular order in which theoperations in FIGS. 6A-6C have been described is merely one example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein.

FIGS. 7A-7C are a flowchart of a method 700 that enables low-visionusers to interact with touch-sensitive secondary displays, in accordancewith some embodiments. The method 700 is performed (702) at a computingsystem including one or more processors, memory, a first housingincluding a primary display, and a second housing at least partiallycontaining a touch-sensitive secondary display that is distinct from theprimary display. Some operations in method 700 are, optionally, combinedand/or the order of some operations is, optionally, changed.

In some embodiments, the computing system is portable computing system100 (FIG. 1A) or desktop computing system 200 (FIGS. 2A-2D). In someembodiments, the primary display is primary display 102 (FIG. 1A) whichis implemented in display portion 110 (also referred to herein as afirst housing 110 that includes the primary display 102) of portablecomputing system 100 (FIG. 1A) and the second housing is a body portion120 of the portable computing system 100, and the second housing atleast partially contains the touch-sensitive secondary display (e.g.,dynamic function row 104, FIGS. 1A-1B) and a physical keyboard (e.g.,the set of physical keys 106) (704).

In some embodiments, the second housing is not connected to the firsthousing (706), e.g., because the first housing is part of a first deviceand the second housing is part of a different device other than thefirst device (e.g., the second housing could be part of a mobile phone,a tablet device, or any other device that includes affordances that aredisplayed on a smaller secondary display while that other device isconnected to a computing system that includes a larger primary display).As one non-limiting example, in some embodiments, the second housing isperipheral keyboard 206 (FIGS. 2A-2B) of desktop computing system 200,which at least partially contains the touch-sensitive secondary display(e.g., dynamic function row 104, FIGS. 2A-2B) and the physical keyboard(e.g., the set of physical keys 106, FIGS. 2A-2B). As another example,in some embodiments, the second housing is first peripheral inputmechanism 212 (FIG. 2C) of desktop computing system 200, which at leastpartially contains the touch-sensitive secondary display (e.g., dynamicfunction row 104, FIG. 2C) and the second housing includes an inputmechanism (e.g., touchpad 108, FIG. 2C) and does not include thephysical keyboard. As one more example, in some embodiments, the secondhousing is part of a wearable computing device (708), such as a smartwatch.

As described below, the method 700 (and associated interfaces) enableslow-vision users to interact with touch-sensitive secondary displays. Ascompared to method 600, method 700 includes operating thetouch-sensitive secondary display in an accessibility mode beforeproviding zoomed-in representations at the primary display. The examplesand descriptions provided above in reference to method 600 are alsoapplicable to method 700 and, for brevity, those examples anddescriptions are generally not repeated here. As shown in FIG. 7A, themethod 700 includes: operating (710) the touch-sensitive secondarydisplay in an accessibility mode. For example, as shown in FIGS. 5A-5B,a system preference option is available that allows users to enable theaccessibility mode for the touch-sensitive secondary display (e.g., byselecting the checkbox to “Enable TouchBar Zoom” using a cursor 504). Insome embodiments, a checkmark option is also presented at thetouch-sensitive secondary display 104 that allows users to enable ordisable accessibility mode by providing an input at the touch-sensitivesecondary display 104 (as shown in FIG. 5A-5B).

The remaining operations of method 700 (operations 714-738) are eachperformed (712) while operating the touch-sensitive secondary display inthe accessibility mode. The method 700 includes displaying (714), on theprimary display, a first user interface for an application (e.g., anexample user interface is shown for an email application in FIG. 5C).The method also includes displaying (716), on the touch-sensitivesecondary display, a second user interface that includes: (i) aplurality of application-specific affordances that control functionsavailable within the application (e.g., affordances 5080-5084, withadditional description provided for these affordances above in referenceto method 600) and (ii) at least one system-level affordance thatcontrols a system-level function (e.g., affordances 5079 and 5085-5088,with additional description provided for these affordances above inreference to method 600). Each of the plurality of application-specificaffordances and the at least one system-level affordance are displayedwith a first display size. In some embodiments, each respectiveaffordance is (only) selectable (718) via one or more inputs at thetouch-sensitive secondary display (additional details regardingoperation 718 are provided above in reference to operation 614).

The method 700 further includes: detecting (720), via thetouch-sensitive secondary display, an input (e.g., input 5102, FIG. 5D)that contacts at least one application-specific affordance of theplurality of application-specific affordances (e.g., affordance 5104).In response to the detecting the input and while the input remains incontact with the touch-sensitive secondary display (722), the methodincludes: continuing to display, on the primary display, the first userinterface for the application and displaying, on the primary display, azoomed-in representation of the at least one application-specificaffordance. The zoomed-in representation of the at least oneapplication-specific affordance is displayed with a second display sizethat is larger than the first display size. Additional details andexamples of operation 722 are provided above in reference to operation618.

In some instances, low-vision users of computing systems rely onmemorized key locations on a keyboard so that they are able toaccurately provide inputs to a computing system. For computing systemsthat include a touch-sensitive secondary display with often-changingaffordances, these users are not able to rely solely on memorization toprovide accurate inputs. Displaying a zoomed-in representation of atleast one affordance of the application-specific affordance improvesoperability of the computing system, because low-vision users are ableto interact with controls available at the touch-sensitive secondarydisplay that may be too small (or may be occluded from view because auser's finger is covering up the displayed controls) for the low-visionusers to view accurately. In this way, low-vision users are able to takeadvantage of an improved man-machine interface by, e.g., havingsustained interactions with a touch-sensitive secondary display (insteadof having to constantly correct erroneous inputs).

Turning now to FIG. 7B, displaying the zoomed-in representation of theat least one application-specific affordance includes (724) displayingthe zoomed-in representation of the at least one application-specificaffordance within a zoomed-in representation of the second userinterface. Additional details and examples regarding operation 724 areprovided above in reference to operation 620. Displaying a zoomed-inview of the at least one application-specific affordance within azoomed-in representation of the second user interface in response to asingle input provides users with clear visual feedback indicating whichaffordance they may be selecting and indicating which affordances arelocated in proximity thereto. Providing this improved visual feedback tothe user enhances operability of the device and makes the human-machineinterface more efficient (e.g., by helping the user to provide properinputs and reducing user mistakes when operating/interacting withaffordances and other neighboring affordances displayed on the secondarydisplay). Additionally, allowing these users to accurately viewaffordances displayed on a small screen enables a sustained interactionwith the touch-sensitive secondary display that would not otherwise bepossible due to frequent mistakes (e.g., incorrect selections) and theneed to waste time correcting these mistakes.

In some embodiments, displaying the zoomed-in representation of thesecond user interface includes (726) displaying a focus indicator withinthe zoomed-in representation of the second user interface that ispositioned based at least in part on a position on the touch-sensitivesecondary display at which the input contacted the touch-sensitivesecondary display. Additional details and examples regarding operation726 are provided above in reference to operation 622. Displaying a focusindicator within the zoomed-in representation of the second userinterface provides users with clear visual feedback as to the locationof their finger on the touch-sensitive secondary display. In someinstances, the users may not be able to see small affordances displayedon the touch-sensitive secondary display due to vision problems or dueto their finger obscuring affordances located underneath. Therefore,providing the focus indicator within the zoomed-in representationenhances operability of the device and makes the human-machine interfacemore efficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with affordances andother neighboring affordances displayed on the secondary display).Additionally, allowing these users to accurately understand the locationof their finger on the touch-sensitive secondary display enables asustained interaction with the touch-sensitive secondary display thatwould not otherwise be possible due to frequent mistakes (e.g.,incorrect selections) and the need to waste time correcting thesemistakes.

In some embodiments, the focus indicator includes (728) a representationof a countdown timer and the method 700 includes: in accordance with adetermination that the input has remained in contact with the at leastone application-specific affordance for more than a predetermined amountof time, the method 700 includes updating the representation of thecountdown timer to indicate that the countdown timer is active. Inaccordance with a determination that the countdown timer has expired,the method 700 includes activating the at least one application-specificaffordance. Additional details and examples regarding operation 728 areprovided above in reference to operation 624. Displaying a focusindicator with a representation of a countdown timer that beginscounting down after an input has remained in contact with an affordancefor more than a predetermined period of time provides users with clearvisual feedback that they are about to select an affordance. In someinstances, the users may not be able to see small affordances displayedon the touch-sensitive secondary display due to vision problems or dueto their finger obscuring affordances located underneath and, thus,these users may not realize when they are activating/selecting variousaffordances. Therefore, providing the focus indicator with the countdowntimer enhances operability of the device and makes the human-machineinterface more efficient (e.g., by helping the user to provide properinputs and reducing user mistakes when operating/interacting withaffordances and other neighboring affordances displayed on the secondarydisplay). Additionally, allowing these users to accurately understandwhen they are about to activate an affordance enables a sustainedinteraction with the touch-sensitive secondary display that would nototherwise be possible due to frequent mistakes (e.g., incorrectselections) and the need to waste time correcting these mistakes.

In some embodiments, the at least one application-specific affordance isassociated (730) with a slider, and activating the at least oneapplication-specific affordance includes updating the zoomed-inrepresentation of the at least one application-specific in accordancewith (or, more generally, to allow) movement of the at least oneapplication-specific affordance along the slider. Additional details andexamples regarding operation 730 are provided above in reference tooperation 626. Activating an affordance that is associated with a sliderafter expiration of a countdown timer helps to enhance operability ofthe device and makes the human-machine interface more efficient (e.g.,by reducing user mistakes when operating/interacting with affordancesand helping to avoid accidental modification of a slider). Additionally,allowing users to accurately understand when they are able to manipulatea slider enables a sustained interaction with the touch-sensitivesecondary display that would not otherwise be possible due to frequentmistakes (e.g., incorrect or accidental manipulations of a slider) andthe need to waste time correcting these mistakes.

In some embodiments, the zoomed-in representation of the at least oneapplication-specific affordance is displayed (732) on the primarydisplay in accordance with a determination that the input has remainedin continuous contact with the touch-sensitive secondary display formore than a predetermined amount of time. Additional details andexamples regarding operation 732 are provided above in reference tooperation 628.

Attention is now directed to FIG. 7C. In some embodiments, the method700 also includes: while the input remains in contact with at least oneapplication-specific affordance at the touch-sensitive secondarydisplay, detecting (736) a tap gesture at the touch-sensitive secondarydisplay that does not contact the at least one application-specificaffordance (e.g., tap 5107, FIG. 5M, or tap 5105, FIG. 5H). In someembodiments, the touch-sensitive secondary display includes a first areacomprising the at least one application-specific affordance and a secondarea comprising other affordances in the plurality ofapplication-specific affordances (i.e., these other affordances do notinclude the at least one application-specific affordance), and the tapgesture is received at the second area and thus does not contact the atleast one application-specific affordance. In response to detecting thetap gesture, the method 700 includes: activating the at least oneapplication-specific affordance. Additional details and examplesregarding operation 736 are provided above in reference to operation632. Allowing activation of an affordance that is in contact with aninput in response to a tap gesture that is not over the affordance (a“split-tap gesture”) enhances operability of the device and makes thehuman-machine interface more efficient (e.g., by allowing users to placea first finger over a desired affordance and then use a different fingerto perform a selection of that desired affordance). Additionally,allowing users to move their first finger freely around thetouch-sensitive secondary display allows users to maintain a sustainedinteraction with the touch-sensitive secondary display (by exploringwhich affordances are displayed at the touch-sensitive secondarydisplay), that would not otherwise be possible due to frequent mistakes(e.g., incorrect or accidental selections of affordances) and the needto waste time correcting these mistakes.

In some embodiments, the method 700 includes: detecting (738), at thetouch-sensitive secondary display, a predefined gesture (such as thoseexample predefined gestures discussed above in reference to operation634) that manipulates a zoom level that is used to display the zoomed-inrepresentation of the at least one application-specific affordance atthe primary display. In response to detecting the predefined gesture (orin response to detecting each incremental manipulation of the zoomlevel), the method 700 includes: updating the zoomed-in representationat the primary display as the zoom level is manipulated using thepredefined gesture. Additional details and examples regarding operation738 are provided above in reference to operation 634. Allowing users tomanipulate a zoom level for the zoomed-in representation using apredefined gesture enhances operability of the device and makes thehuman-machine interface more efficient (e.g., by allowing users toquickly and easily adjust the zoom level to suit their personalpreferences). Additionally, allowing users to manipulate the zoom levelallows users to maintain a sustained interaction with thetouch-sensitive secondary display by ensuring that these users are ableto adjust the zoom level so that they are able to accurately viewaffordances that may be displayed at different display sizes, which isimportant for low-vision users of various sight levels.

It should be understood that the particular order in which theoperations in FIGS. 7A-7C have been described is merely one example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein.

FIGS. 8A-8C are a flowchart of a method 800 that enables low-visionusers to interact with touch-sensitive secondary displays, in accordancewith some embodiments. The method 800 is performed (802) at a computingsystem including one or more processors, memory, a first housingincluding a primary display, and a second housing at least partiallycontaining a touch-sensitive secondary display that is distinct from theprimary display. Some operations in method 800 are, optionally, combinedand/or the order of some operations is, optionally, changed.

In some embodiments, the computing system is portable computing system100 (FIG. 1A) or desktop computing system 200 (FIGS. 2A-2D). In someembodiments, the primary display is primary display 102 (FIG. 1A) whichis implemented in display portion 110 (also referred to herein as afirst housing 110 that includes the primary display 102) of portablecomputing system 100 (FIG. 1A) and the second housing is a body portion120 of the portable computing system 100, and the second housing atleast partially contains the touch-sensitive secondary display (e.g.,dynamic function row 104, FIGS. 1A-1B) and a physical keyboard (e.g.,the set of physical keys 106) (804).

In some embodiments, the second housing is not connected to the firsthousing (806), e.g., because the first housing is part of a first deviceand the second housing is part of a different device other than thefirst device (e.g., the second housing could be part of a mobile phone,a tablet device, or any other device that includes affordances that aredisplayed on a smaller secondary display while that other device isconnected to a computing system that includes a larger primary display).As one non-limiting example, in some embodiments, the second housing isperipheral keyboard 206 (FIGS. 2A-2B) of desktop computing system 200,which at least partially contains the touch-sensitive secondary display(e.g., dynamic function row 104, FIGS. 2A-2B) and the physical keyboard(e.g., the set of physical keys 106, FIGS. 2A-2B). As another example,in some embodiments, the second housing is first peripheral inputmechanism 212 (FIG. 2C) of desktop computing system 200, which at leastpartially contains the touch-sensitive secondary display (e.g., dynamicfunction row 104, FIG. 2C) and the second housing includes an inputmechanism (e.g., touchpad 108, FIG. 2C) and does not include thephysical keyboard. As one more example, in some embodiments, the secondhousing is part of a wearable computing device (808), such as a smartwatch.

As described below, the method 800 (and associated interfaces) enableslow-vision users to interact with touch-sensitive secondary displays.The examples and descriptions provided above in reference to methods 600and 700 are also applicable to method 800 and, for brevity, thoseexamples and descriptions are generally not repeated here. As shown inFIG. 8A, the method 800 includes displaying (802), on the primarydisplay, a first user interface for an application (e.g., user interfacefor an email application, FIG. 5C). The method 800 also includesdisplaying (812), on the touch-sensitive secondary display, a seconduser interface that includes: (i) a plurality of application-specificaffordances that control functions available within the application(e.g., affordances 5080-5084, with additional description provided forthese affordances above in reference to method 600) and (ii) at leastone system-level affordance that controls a system-level function (e.g.,affordances 5079 and 5085-5088, with additional description provided forthese affordances above in reference to method 600). In someembodiments, each respective affordance is (only) selectable (814) viaone or more inputs at the touch-sensitive secondary display (additionaldetails regarding operation 814 are provided above in reference tooperation 614).

In some embodiments, the first application-specific affordance isdisplayed at the touch-sensitive secondary display with a first displaysize, and the method 800 further includes (818): in response todetecting the first input and while the first input remains in contactwith the first application-specific affordance: continuing to display,on the primary display, the first user interface for the application;and displaying a zoomed-in representation of the firstapplication-specific affordance on the primary display, wherein thezoomed-in representation is displayed with a second display size that islarger than the first display size. Additional details and examplesregarding operation 818 are provided above in reference to operation618.

The method 800 additionally includes: detecting, via the touch-sensitivesecondary display, a first input over a first application-specificaffordance of the plurality of application-specific affordances (e.g.,input 5102 over affordance 5089, FIG. 5M). While the first input remainsin contact with the first application-specific affordance, the method800 includes (820): detecting, via the touch-sensitive secondarydisplay, a second input that is not over the first application-specificaffordance (e.g., input 5107 that is not over the affordance 5089, FIG.5M) and in response to detecting the second input, activating the firstapplication-specific affordance (e.g., bolding selected text on theprimary display due to activation of the affordance 5089, FIG. 5N).Additional examples and descriptions of operation 820 are provided abovein reference to operation 632. Allowing activation of an affordance thatis in contact with an input in response to a tap gesture that is notover the affordance (a “split-tap gesture”) enhances operability of thedevice and makes the human-machine interface more efficient (e.g., byallowing users to place a first finger over a desired affordance andthen use a different finger to perform a selection of that desiredaffordance, thereby ensuring that only the desired affordance isactivated and helping to minimize erroneous selections/activations).Additionally, allowing users to move their first finger freely aroundthe touch-sensitive secondary display (without selecting affordances)allows users to maintain a sustained interaction with thetouch-sensitive secondary display (by exploring which affordances aredisplayed at the touch-sensitive secondary display), that would nototherwise be possible due to frequent mistakes (e.g., incorrect oraccidental selections of affordances) and the need to waste timecorrecting these mistakes.

Attention is now directed to FIG. 8B. In some embodiments, displayingthe zoomed-in representation of the at least one application-specificaffordance includes (822) displaying the zoomed-in representation of theat least one application-specific affordance within a zoomed-inrepresentation of the second user interface. Additional details andexamples regarding operation 822 are provided above in reference tooperation 620. Displaying a zoomed-in view of the at least oneapplication-specific affordance within a zoomed-in representation of thesecond user interface in response to a single input provides users withclear visual feedback indicating which affordance they may be selectingand indicating which affordances are located in proximity thereto.Providing this improved visual feedback to the user enhances operabilityof the device and makes the human-machine interface more efficient(e.g., by helping the user to provide proper inputs and reducing usermistakes when operating/interacting with affordances and otherneighboring affordances displayed on the secondary display).Additionally, allowing these users to accurately view affordancesdisplayed on a small screen enables a sustained interaction with thetouch-sensitive secondary display that would not otherwise be possibledue to frequent mistakes (e.g., incorrect selections) and the need towaste time correcting these mistakes.

In some embodiments, displaying the zoomed-in representation of thesecond user interface includes (824) displaying a focus indicator withinthe zoomed-in representation of the second user interface that ispositioned based at least in part on a position on the touch-sensitivesecondary display at which the input contacted the touch-sensitivesecondary display. Additional details and examples regarding operation824 are provided above in reference to operation 622. Displaying a focusindicator within the zoomed-in representation of the second userinterface provides users with clear visual feedback as to the locationof their finger on the touch-sensitive secondary display. In someinstances, the users may not be able to see small affordances displayedon the touch-sensitive secondary display due to vision problems or dueto their finger obscuring affordances located underneath. Therefore,providing the focus indicator within the zoomed-in representationenhances operability of the device and makes the human-machine interfacemore efficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with affordances andother neighboring affordances displayed on the secondary display).Additionally, allowing these users to accurately understand the locationof their finger on the touch-sensitive secondary display enables asustained interaction with the touch-sensitive secondary display thatwould not otherwise be possible due to frequent mistakes (e.g.,incorrect selections) and the need to waste time correcting thesemistakes.

In some embodiments, the at least one application-specific affordance isassociated (828) with a slider, and activating the at least oneapplication-specific affordance includes updating the zoomed-inrepresentation of the at least one application-specific in accordancewith (or, more generally, to allow) movement of the at least oneapplication-specific affordance along the slider. Additional details andexamples regarding operation 828 are provided above in reference tooperation 626. Activating an affordance that is associated with a sliderafter expiration of a countdown timer helps to enhance operability ofthe device and makes the human-machine interface more efficient (e.g.,by reducing user mistakes when operating/interacting with affordancesand helping to avoid accidental modification of a slider). Additionally,allowing users to accurately understand when they are able to manipulatea slider enables a sustained interaction with the touch-sensitivesecondary display that would not otherwise be possible due to frequentmistakes (e.g., incorrect or accidental manipulations of a slider) andthe need to waste time correcting these mistakes.

In some embodiments, the zoomed-in representation of the at least oneapplication-specific affordance is displayed (830) on the primarydisplay in accordance with a determination that the input has remainedin continuous contact with the touch-sensitive secondary display formore than a predetermined amount of time. Additional details andexamples regarding operation 830 are provided above in reference tooperation 628.

In some embodiments, the zoomed-in representation of the at least oneapplication-specific affordance is displayed in accordance with adetermination that the touch-sensitive secondary display is operating inan accessibility mode. Additional details and examples regardingoperation 832 are provided above in reference to operations 630, 710,and 712.

Turning now to FIG. 8C, in some embodiments, the method 800 includes:detecting (834), at the touch-sensitive secondary display, a predefinedgesture (e.g., examples are provided above in reference to operation634) that manipulates a zoom level that is used to display the zoomed-inrepresentation of the at least one application-specific affordance atthe primary display; and in response to detecting the predefinedgesture, updating the zoomed-in representation at the primary display asthe zoom level is manipulated using the predefined gesture. Additionaldetails and examples regarding operation 834 are provided above inreference to operation 634. Allowing users to manipulate a zoom levelfor the zoomed-in representation using a predefined gesture enhancesoperability of the device and makes the human-machine interface moreefficient (e.g., by allowing users to quickly and easily adjust the zoomlevel to suit their personal preferences). Additionally, allowing usersto manipulate the zoom level allows users to maintain a sustainedinteraction with the touch-sensitive secondary display by ensuring thatthese users are able to adjust the zoom level so that they are able toaccurately view affordances that may be displayed at different displaysizes, which is important for low-vision users of various sight levels.

It should be understood that the particular order in which theoperations in FIGS. 8A-8C have been described is merely one example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein.

In accordance with some embodiments, FIG. 9 shows a functional blockdiagram of a computing system 900 (also referred to, in accordance withsome embodiments, as an electronic device 900) configured in accordancewith the principles of the various described embodiments. The functionalblocks of the system are, optionally, implemented by hardware, software,firmware, or a combination thereof to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 9 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein. For ease ofdiscussion, the computing system 900 is implemented as a portablecomputing system 100 (FIG. 1A). In some embodiments, the computingsystem 900 is implemented in accordance with any of the devices/systemsshown in FIGS. 1A-2D.

As shown in FIG. 9, the computing system 900, includes a primary displayunit 902 configured to display information (e.g., touch-sensitivedisplay system 112, also referred to as a primary touch screen, primarytouch-sensitive display, and primary touch screen display, FIG. 1A), atouch-sensitive secondary display unit 904 configured to receivecontacts, gestures, and other user inputs on the touch-sensitivesecondary display, and a processing unit 910. The system 900 optionallyincludes a physical keyboard unit 905 configured to receive keyboardinputs, and, in some embodiments, the optional physical keyboard unit isincluded with the touch-sensitive secondary display unit 904 in a secondhousing that is distinct from a first housing that includes the primarydisplay unit 902. In some embodiments the first and second housings arerotatably connected (e.g., for embodiments in which the computing system900 is a laptop computer) and in other embodiments, the first and secondhousings are not connected and may be part of separate computing devicesthat form the system 900 (e.g., the first housing is part of a laptopcomputer and the second housing is part of a separate device such as awearable computing device, like a smart watch). The processing unit 910is coupled with the primary display unit 902, the touch-sensitivesecondary display unit 904, and optionally the physical keyboard unit905. In some embodiments, the processing unit 910 includes a displayingunit (e.g., displaying unit 912), a detecting unit 914 (e.g., detectingunit 914), and an affordance activating unit (e.g., affordanceactivating unit 916).

The processing unit is configured to: display (e.g., with the displayingunit 912 in conjunction with the primary display unit 902), on theprimary display, a first user interface for an application; display(e.g., with the displaying unit 912 in conjunction with thetouch-sensitive secondary display unit 904), on the touch-sensitivesecondary display, a second user interface that includes a plurality ofapplication-specific affordances that control functions available withinthe application, and each of the plurality of application-specificaffordances is displayed with a first display size; detect (e.g., withthe detecting unit 914 in conjunction with the touch-sensitive secondarydisplay unit 904), via the touch-sensitive secondary display, an inputthat contacts at least one application-specific affordance of theplurality of application-specific affordances; and in response todetecting the input and while the input remains in contact with thetouch-sensitive secondary display: (i) continue to display (e.g., withthe displaying unit 912 in conjunction with the primary display unit902), on the primary display, the first user interface for theapplication and (ii) display (e.g., with the displaying unit 912 inconjunction with the primary display unit 902), on the primary display,a zoomed-in representation of the at least one application-specificaffordance, and the zoomed-in representation of the at least oneapplication-specific affordance is displayed with a second display sizethat is larger than the first display size.

In accordance with some embodiments of the computing system 900,displaying the zoomed-in representation of the at least oneapplication-specific affordance includes displaying the zoomed-inrepresentation of the at least one application-specific affordancewithin a zoomed-in representation of the second user interface.

In accordance with some embodiments of the computing system 900,displaying the zoomed-in representation of the second user interfaceincludes displaying a focus indicator within the zoomed-inrepresentation of the second user interface that is positioned based atleast in part on a position on the touch-sensitive secondary display atwhich the input contacted the touch-sensitive secondary display.

In accordance with some embodiments of the computing system 900, thefocus indicator includes a representation of a countdown timer and theprocessing unit is further configured to: in accordance with adetermination that the input has remained in contact with the at leastone application-specific affordance for more than a predetermined amountof time, update the representation of the countdown timer to indicatethat the countdown timer is active (e.g., with the displaying unit 912in conjunction with the primary display unit 902); and in accordancewith a determination that the countdown timer has expired, activate theat least one application-specific affordance (e.g., with the affordanceactivating unit 916).

In accordance with some embodiments of the computing system 900, the atleast one application-specific affordance is associated with a slider,and activating the at least one application-specific affordance includesupdating the zoomed-in representation of the at least oneapplication-specific in accordance with (or, more generally, to allow)movement of the at least one application-specific affordance along theslider

In accordance with some embodiments of the computing system 900, theprocessing unit is further configured to: while the input remains incontact with the at least one application-specific affordance at thetouch-sensitive secondary display, detect a tap gesture at thetouch-sensitive secondary display (e.g., with the detecting unit 914 inconjunction with the touch-sensitive secondary display unit 904) thatdoes not contact the at least one application-specific affordance; andin response to detecting the tap gesture, activate the at least oneapplication-specific affordance (e.g., with the affordance activatingunit 916). In some embodiments, the touch-sensitive secondary displayincludes a first area comprising the at least one application-specificaffordance and a second area comprising other affordances in theplurality of application-specific affordances (i.e., these otheraffordances do not include the at least one application-specificaffordance), and the tap gesture is received at the second area and thusdoes not contact the at least one application-specific affordance.

In accordance with some embodiments of the computing system 900, theprocessing unit is further configured to: detect (e.g., with thedetecting unit 914 in conjunction with the touch-sensitive secondarydisplay unit 904), at the touch-sensitive secondary display, apredefined gesture that manipulates a zoom level that is used to displaythe zoomed-in representation of the at least one application-specificaffordance at the primary display; and in response to detecting thepredefined gesture, update the zoomed-in representation at the primarydisplay as the zoom level is manipulated using the predefined gesture(e.g., with the displaying unit 912 in conjunction with the primarydisplay unit 902).

In accordance with some embodiments of the computing system 900, thezoomed-in representation of the at least one application-specificaffordance is displayed on the primary display in accordance with adetermination that the input has remained in continuous contact with thetouch-sensitive secondary display for more than a predetermined amountof time.

In accordance with some embodiments of the computing system 900, thezoomed-in representation of the at least one application-specificaffordance is displayed in accordance with a determination that thetouch-sensitive secondary display is operating in an accessibility mode.

In accordance with some embodiments of the computing system 900, each ofthe plurality of application-specific affordances is selectable via oneor more inputs at the touch-sensitive secondary display.

In accordance with some embodiments of the computing system 900, thesecond housing also at least partially contains a physical keyboard.

In accordance with some embodiments of the computing system 900, thesecond housing is not connected to the first housing.

In accordance with some embodiments of the computing system 900, thesecond housing is part of a wearable computing device.

In accordance with some embodiments, FIG. 10 shows a functional blockdiagram of a computing system 1000 (also referred to, in accordance withsome embodiments, as an electronic device 1000) configured in accordancewith the principles of the various described embodiments. The functionalblocks of the system are, optionally, implemented by hardware, software,firmware, or a combination thereof to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 10 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein. For ease ofdiscussion, the computing system 1000 is implemented as a portablecomputing system 100 (FIG. 1A). In some embodiments, the computingsystem 1000 is implemented in accordance with any of the devices/systemsshown in FIGS. 1A-2D.

As shown in FIG. 10, the computing system 1000, includes a primarydisplay unit 1002 configured to display information (e.g.,touch-sensitive display system 112, also referred to as a primary touchscreen, primary touch-sensitive display, and primary touch screendisplay, FIG. 1A), a touch-sensitive secondary display unit 1004configured to receive contacts, gestures, and other user inputs on thetouch-sensitive secondary display, and a processing unit 1010. Thesystem 1000 optionally includes a physical keyboard unit 1005 configuredto receive keyboard inputs, and, in some embodiments, the optionalphysical keyboard unit is included with the touch-sensitive secondarydisplay unit 1004 in a second housing that is distinct from a firsthousing that includes the primary display unit 1002. In some embodimentsthe first and second housings are rotatably connected (e.g., forembodiments in which the computing system 1000 is a laptop computer) andin other embodiments, the first and second housings are not connectedand may be part of separate computing devices that form the system 1000(e.g., the first housing is part of a laptop computer and the secondhousing is part of a separate device such as a wearable computingdevice, like a smart watch). The processing unit 1010 is coupled withthe primary display unit 1002, the touch-sensitive secondary displayunit 1004, and optionally the physical keyboard unit 1005. In someembodiments, the processing unit 1010 includes a displaying unit (e.g.,displaying unit 1012), a detecting unit 1014 (e.g., detecting unit1014), an affordance activating unit (e.g., affordance activating unit1016), and an accessibility mode operating unit (e.g., accessibilitymode operating unit 1018).

The processing unit is configured to: operate the touch-sensitivesecondary display in an accessibility mode; while operating thetouch-sensitive secondary display in the accessibility mode: display(e.g., with the displaying unit 1012 in conjunction with the primarydisplay unit 1002), on the primary display, a first user interface foran application; and display (e.g., with the displaying unit 1012 inconjunction with the touch-sensitive secondary display unit 1004), onthe touch-sensitive secondary display, a second user interface thatincludes: (i) a plurality of application-specific affordances thatcontrol functions available within the application and (ii) at least onesystem-level affordance that controls a system-level function, and eachof the plurality of application-specific affordances and the at leastone system-level affordance are displayed with a first display size;detect (e.g., with the detecting unit 1014 in conjunction with thetouch-sensitive secondary display unit 1004), via the touch-sensitivesecondary display, an input that contacts at least oneapplication-specific affordance of the plurality of application-specificaffordances; and in response to the detecting the input and while theinput remains in contact with the touch-sensitive secondary display:continue to display (e.g., with the displaying unit 1012 in conjunctionwith the primary display unit 1002), on the primary display, the firstuser interface for the application and display (e.g., with thedisplaying unit 1012 in conjunction with the primary display unit 1002),on the primary display, a zoomed-in representation of the at least oneapplication-specific affordance, and the zoomed-in representation of theat least one application-specific affordance is displayed with a seconddisplay size that is larger than the first display size.

In accordance with some embodiments of the computing system 1000,displaying the zoomed-in representation of the at least oneapplication-specific affordance includes displaying the zoomed-inrepresentation of the at least one application-specific affordancewithin a zoomed-in representation of the second user interface.

In accordance with some embodiments of the computing system 1000,displaying the zoomed-in representation of the second user interfaceincludes displaying a focus indicator within the zoomed-inrepresentation of the second user interface that is positioned based atleast in part on a position on the touch-sensitive secondary display atwhich the input contacted the touch-sensitive secondary display.

In accordance with some embodiments of the computing system 1000, thefocus indicator includes a representation of a countdown timer and theprocessing unit is further configured to: in accordance with adetermination that the input has remained in contact with the at leastone application-specific affordance for more than a predetermined amountof time, update the representation of the countdown timer to indicatethat the countdown timer is active (e.g., with the displaying unit 1012in conjunction with the primary display unit 1002); and in accordancewith a determination that the countdown timer has expired, activate theat least one application-specific affordance (e.g., with the affordanceactivating unit 1016).

In accordance with some embodiments of the computing system 1000, the atleast one application-specific affordance is associated with a slider,and activating the at least one application-specific affordance includesupdating the zoomed-in representation of the at least oneapplication-specific in accordance with (or, more generally, to allow)movement of the at least one application-specific affordance along theslider.

In accordance with some embodiments of the computing system 1000, theprocessing unit is further configured to: while the input remains incontact with at least one application-specific affordance at thetouch-sensitive secondary display, detect a tap gesture at thetouch-sensitive secondary display that does not contact the at least oneapplication-specific affordance (e.g., with the detecting unit 1012 inconjunction with the touch-sensitive secondary display unit 1004); andin response to detecting the tap gesture, activate the at least oneapplication-specific affordance (e.g., with the affordance activatingunit 1016). In some embodiments, the touch-sensitive secondary displayincludes a first area comprising the at least one application-specificaffordance and a second area comprising other affordances in theplurality of application-specific affordances (i.e., these otheraffordances do not include the at least one application-specificaffordance), and the tap gesture is received at the second area and thusdoes not contact the at least one application-specific affordance.

In accordance with some embodiments of the computing system 1000, theprocessing unit is further configured to: detect (e.g., with thedetecting unit 1012 in conjunction with the touch-sensitive secondarydisplay unit 1004), with the, at the touch-sensitive secondary display,a predefined gesture that manipulates a zoom level that is used todisplay the zoomed-in representation of the at least oneapplication-specific affordance at the primary display; and in responseto detecting the predefined gesture, update the zoomed-in representationat the primary display as the zoom level is manipulated using thepredefined gesture (e.g., with the displaying unit 1012 in conjunctionwith the primary display unit 1002).

In accordance with some embodiments of the computing system 1000, thezoomed-in representation of the at least one application-specificaffordance is displayed on the primary display in accordance with adetermination that the input has remained in continuous contact with thetouch-sensitive secondary display for more than a predetermined amountof time.

In accordance with some embodiments of the computing system 1000, eachof the plurality of application-specific affordances is selectable viaone or more inputs at the touch-sensitive secondary display.

In accordance with some embodiments of the computing system 1000, thesecond housing also at least partially contains a physical keyboard.

In accordance with some embodiments of the computing system 1000, thesecond housing is not connected to the first housing.

In accordance with some embodiments of the computing system 1000, thesecond housing is part of a wearable computing device.

In accordance with some embodiments, FIG. 11 shows a functional blockdiagram of a computing system 1100 (also referred to, in accordance withsome embodiments, as an electronic device 1100) configured in accordancewith the principles of the various described embodiments. The functionalblocks of the system are, optionally, implemented by hardware, software,firmware, or a combination thereof to carry out the principles of thevarious described embodiments. It is understood by persons of skill inthe art that the functional blocks described in FIG. 11 are, optionally,combined or separated into sub-blocks to implement the principles of thevarious described embodiments. Therefore, the description hereinoptionally supports any possible combination or separation or furtherdefinition of the functional blocks described herein. For ease ofdiscussion, the computing system 1100 is implemented as a portablecomputing system 100 (FIG. 1A). In some embodiments, the computingsystem 1100 is implemented in accordance with any of the devices/systemsshown in FIGS. 1A-2D.

As shown in FIG. 11, the computing system 1100, includes a primarydisplay unit 1102 configured to display information (e.g.,touch-sensitive display system 112, also referred to as a primary touchscreen, primary touch-sensitive display, and primary touch screendisplay, FIG. 1A), a touch-sensitive secondary display unit 1104configured to receive contacts, gestures, and other user inputs on thetouch-sensitive secondary display, and a processing unit 1110. Thesystem 1100 optionally includes a physical keyboard unit 1105 configuredto receive keyboard inputs, and, in some embodiments, the optionalphysical keyboard unit is included with the touch-sensitive secondarydisplay unit 1104 in a second housing that is distinct from a firsthousing that includes the primary display unit 1102. In some embodimentsthe first and second housings are rotatably connected (e.g., forembodiments in which the computing system 1100 is a laptop computer) andin other embodiments, the first and second housings are not connectedand may be part of separate computing devices that form the system 1100(e.g., the first housing is part of a laptop computer and the secondhousing is part of a separate device such as a wearable computingdevice, like a smart watch). The processing unit 1110 is coupled withthe primary display unit 1102, the touch-sensitive secondary displayunit 1104, and optionally the physical keyboard unit 1105. In someembodiments, the processing unit 1110 includes a displaying unit (e.g.,displaying unit 1112), a detecting unit 1114 (e.g., detecting unit1114), and an affordance activating unit (e.g., affordance activatingunit 1116).

The processing unit is configured to: display (e.g., with the displayingunit 1112 in conjunction with the primary display unit 1102), on theprimary display, a first user interface for an application; display(e.g., with the displaying unit 1112 in conjunction with thetouch-sensitive secondary display unit 1104), on the touch-sensitivesecondary display, a second user interface that includes: (i) aplurality of application-specific affordances that control functionsavailable within the application and (ii) at least one system-levelaffordance that controls a system-level function; detect (e.g., with thedetecting unit 1114 in conjunction with the touch-sensitive secondarydisplay unit 1104), via the touch-sensitive secondary display, a firstinput over a first application-specific affordance of the plurality ofapplication-specific affordances; and while the first input remains incontact with the first application-specific affordance: detect (e.g.,with the detecting unit 1114 in conjunction with the touch-sensitivesecondary display unit 1104), via the touch-sensitive secondary display,a second input that is not over the first application-specificaffordance and in response to detecting the second input, activate thefirst application-specific affordance (e.g., with the affordanceactivating unit 1116).

In accordance with some embodiments of the computing system 1100, theprocessing unit is further configured to: in response to detecting thefirst input and while the first input remains in contact with the firstapplication-specific affordance: continue to display (e.g., with thedisplaying unit 1112 in conjunction with the primary display unit 1102),on the primary display, the first user interface for the application;and display (e.g., with the displaying unit 1112 in conjunction with theprimary display unit 1102) a zoomed-in representation of the firstapplication-specific affordance on the primary display, wherein thezoomed-in representation is displayed with a second display size that islarger than the first display size.

In accordance with some embodiments of the computing system 1100,displaying the zoomed-in representation of the at least oneapplication-specific affordance includes displaying the zoomed-inrepresentation of the at least one application-specific affordancewithin a zoomed-in representation of the second user interface.

In accordance with some embodiments of the computing system 1100,displaying the zoomed-in representation of the second user interfaceincludes displaying a focus indicator within the zoomed-inrepresentation of the second user interface that is positioned based atleast in part on a position on the touch-sensitive secondary display atwhich the input contacted the touch-sensitive secondary display.

In accordance with some embodiments of the computing system 1100, the atleast one application-specific affordance is associated with a slider,and activating the at least one application-specific affordance includesupdating the zoomed-in representation of the at least oneapplication-specific in accordance with (or, more generally, to allow)movement of the at least one application-specific affordance along theslider.

In accordance with some embodiments of the computing system 1100, theprocessing unit is further configured to: detect (e.g., with thedetecting unit 1114 in conjunction with the touch-sensitive secondarydisplay unit 1104), at the touch-sensitive secondary display, apredefined gesture that manipulates a zoom level that is used to displaythe zoomed-in representation of the at least one application-specificaffordance at the primary display; and in response to detecting thepredefined gesture, update the zoomed-in representation at the primarydisplay as the zoom level is manipulated using the predefined gesture(e.g., with the displaying unit 1112 in conjunction with the primarydisplay unit 1102).

In accordance with some embodiments of the computing system 1100, thezoomed-in representation of the at least one application-specificaffordance is displayed on the primary display in accordance with adetermination that the input has remained in continuous contact with thetouch-sensitive secondary display for more than a predetermined amountof time.

In accordance with some embodiments of the computing system 1100, thezoomed-in representation of the at least one application-specificaffordance is displayed in accordance with a determination that thetouch-sensitive secondary display is operating in an accessibility mode.

In accordance with some embodiments of the computing system 1100, eachof the plurality of application-specific affordances and the at leastone system-level affordance are selectable via one or more inputs at thetouch-sensitive secondary display.

In accordance with some embodiments of the computing system 1100, thesecond housing also at least partially contains a physical keyboard.

In accordance with some embodiments of the computing system 1100, thesecond housing is not connected to the first housing.

In accordance with some embodiments of the computing system 1100, thesecond housing is part of a wearable computing device.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best use the invention and variousdescribed embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method, comprising: at a computing systemcomprising one or more processors, a first housing that includes aprimary display, memory, and a second housing at least partiallycontaining a touch-sensitive secondary display that is distinct from theprimary display: operating the touch-sensitive secondary display in anaccessibility mode; while operating the touch-sensitive secondarydisplay in the accessibility mode: displaying, on the primary display, afirst user interface for an application; displaying, on thetouch-sensitive secondary display, a second user interface thatincludes: (i) a plurality of application-specific affordances thatcontrol functions available within the application and (ii) at least onesystem-level affordance that controls a system-level function, whereineach of the plurality of application-specific affordances and the atleast one system-level affordance are displayed with a first displaysize; detecting, via the touch-sensitive secondary display, an inputthat contacts at least one application-specific affordance of theplurality of application-specific affordances; and in response to thedetecting the input and while the input remains in contact with thetouch-sensitive secondary display: continuing to display, on the primarydisplay, the first user interface for the application; displaying, onthe primary display, a zoomed-in representation of the at least oneapplication-specific affordance, wherein the zoomed-in representation ofthe at least one application-specific affordance is displayed with asecond display size that is larger than the first display size.
 2. Themethod of claim 1, wherein displaying the zoomed-in representation ofthe at least one application-specific affordance includes displaying thezoomed-in representation of the at least one application-specificaffordance within a zoomed-in representation of the second userinterface.
 3. The method of claim 2, wherein displaying the zoomed-inrepresentation of the second user interface includes displaying a focusindicator within the zoomed-in representation of the second userinterface that is positioned based at least in part on a position on thetouch-sensitive secondary display at which the input contacted thetouch-sensitive secondary display.
 4. The method of claim 3, wherein thefocus indicator includes a representation of a countdown timer and themethod includes: in accordance with a determination that the input hasremained in contact with the at least one application-specificaffordance for more than a predetermined amount of time, updating therepresentation of the countdown timer to indicate that the countdowntimer is active; and in accordance with a determination that thecountdown timer has expired, activating the at least oneapplication-specific affordance.
 5. The method of claim 4, wherein: theat least one application-specific affordance is associated with aslider, and activating the at least one application-specific affordanceincludes updating the zoomed-in representation of the at least oneapplication-specific affordance in accordance with movement of the atleast one application-specific affordance along the slider.
 6. Themethod of claim 1, wherein the touch-sensitive secondary displayincludes a first area comprising the at least one application-specificaffordance and a second area comprising other affordances distinct fromthe at least one application-specific affordance in the plurality ofapplication-specific affordances, and the method further comprises:while the input remains in contact with at least oneapplication-specific affordance at the first area of the touch-sensitivesecondary display, detecting a tap gesture at the second area of thetouch-sensitive secondary display; and in response to detecting the tapgesture, activating the at least one application-specific affordance. 7.The method of claim 1, further comprising: detecting, at thetouch-sensitive secondary display, a predefined gesture that manipulatesa zoom level that is used to display the zoomed-in representation of theat least one application-specific affordance at the primary display; andin response to detecting the predefined gesture, updating the zoomed-inrepresentation at the primary display as the zoom level is manipulatedusing the predefined gesture.
 8. The method of claim 1, wherein thezoomed-in representation of the at least one application-specificaffordance is displayed on the primary display in accordance with adetermination that the input has remained in continuous contact with thetouch-sensitive secondary display for more than a predetermined amountof time.
 9. The method of claim 1, wherein each of the plurality ofapplication-specific affordances and the at least one system-levelaffordance are selectable via one or more inputs at the touch-sensitivesecondary display.
 10. The method of claim 1, wherein the second housingalso at least partially contains a physical keyboard.
 11. The method ofclaim 1, wherein the second housing is not connected to the firsthousing.
 12. The method of claim 1, wherein the second housing is partof a wearable computing device.
 13. A non-transitory computer-readablestorage medium storing executable instructions that, when executed by acomputing system with one or more processors, a first housing thatincludes a primary display, and a second housing at least partiallycontaining a touch-sensitive secondary display that is distinct from theprimary display, cause the computing system to: operate thetouch-sensitive secondary display in an accessibility mode; whileoperating the touch-sensitive secondary display in the accessibilitymode: display, on the primary display, a first user interface for anapplication; display, on the touch-sensitive secondary display, a seconduser interface that includes: (i) a plurality of application-specificaffordances that control functions available within the application and(ii) at least one system-level affordance that controls a system-levelfunction, wherein each of the plurality of application-specificaffordances and the at least one system-level affordance are displayedwith a first display size; detect, via the touch-sensitive secondarydisplay, an input that contacts at least one application-specificaffordance of the plurality of application-specific affordances; and inresponse to the detecting the input and while the input remains incontact with the touch-sensitive secondary display: continue to display,on the primary display, the first user interface for the application;display, on the primary display, a zoomed-in representation of the atleast one application-specific affordance, wherein the zoomed-inrepresentation of the at least one application-specific affordance isdisplayed with a second display size that is larger than the firstdisplay size.
 14. The non-transitory computer-readable storage medium ofclaim 13, wherein displaying the zoomed-in representation of the atleast one application-specific affordance includes displaying thezoomed-in representation of the at least one application-specificaffordance within a zoomed-in representation of the second userinterface.
 15. The non-transitory computer-readable storage medium ofclaim 14, wherein displaying the zoomed-in representation of the seconduser interface includes displaying a focus indicator within thezoomed-in representation of the second user interface that is positionedbased at least in part on a position on the touch-sensitive secondarydisplay at which the input contacted the touch-sensitive secondarydisplay.
 16. The non-transitory computer-readable storage medium ofclaim 15, wherein the focus indicator includes a representation of acountdown timer and the executable instructions also cause the computingsystem to: in accordance with a determination that the input hasremained in contact with the at least one application-specificaffordance for more than a predetermined amount of time, update therepresentation of the countdown timer to indicate that the countdowntimer is active; and in accordance with a determination that thecountdown timer has expired, activate the at least oneapplication-specific affordance.
 17. A computing system, comprising: oneor more processors; a first housing that includes a primary display; asecond housing at least partially containing a touch-sensitive secondarydisplay that is distinct from the primary display; and memory storingone or more programs that are configured for execution by the one ormore processors, the one or more programs including instructions for:operating the touch-sensitive secondary display in an accessibilitymode; while operating the touch-sensitive secondary display in theaccessibility mode: displaying, on the primary display, a first userinterface for an application; displaying, on the touch-sensitivesecondary display, a second user interface that includes: (i) aplurality of application-specific affordances that control functionsavailable within the application and (ii) at least one system-levelaffordance that controls a system-level function, wherein each of theplurality of application-specific affordances and the at least onesystem-level affordance are displayed with a first display size;detecting, via the touch-sensitive secondary display, an input thatcontacts at least one application-specific affordance of the pluralityof application-specific affordances; and in response to the detectingthe input and while the input remains in contact with thetouch-sensitive secondary display: continuing to display, on the primarydisplay, the first user interface for the application; displaying, onthe primary display, a zoomed-in representation of the at least oneapplication-specific affordance, wherein the zoomed-in representation ofthe at least one application-specific affordance is displayed with asecond display size that is larger than the first display size.
 18. Thecomputing system of claim 17, wherein displaying the zoomed-inrepresentation of the at least one application-specific affordanceincludes displaying the zoomed-in representation of the at least oneapplication-specific affordance within a zoomed-in representation of thesecond user interface.
 19. The computing system of claim 18, whereindisplaying the zoomed-in representation of the second user interfaceincludes displaying a focus indicator within the zoomed-inrepresentation of the second user interface that is positioned based atleast in part on a position on the touch-sensitive secondary display atwhich the input contacted the touch-sensitive secondary display.
 20. Thecomputing system of claim 19, wherein the focus indicator includes arepresentation of a countdown timer and the one or more programs alsoinclude instructions for: in accordance with a determination that theinput has remained in contact with the at least one application-specificaffordance for more than a predetermined amount of time, updating therepresentation of the countdown timer to indicate that the countdowntimer is active; and in accordance with a determination that thecountdown timer has expired, activating the at least oneapplication-specific affordance.